Machine Learning with Windows Azure

This is a brief overview of Machine Learning usage on Azure cloud.

Login to your Azure account, select Machine Learning and follow these

Steps:

1. Create new Experiment
2. Drag-n-drop datasets and modules and modify their properties
3. Create a workflow. There are available pre-configured ML modules with R and Python scripts that:
1. Following best practices may help to solve a problem for specific domain.
2. Have pre-built data schemas.
3. Contain domain specific data processing and feature engineering.
4. Include training algorithms and calculation metrics.

Workflow Concerns:

• A Single Experiment:
• could be too complex
• difficult to navigate
• easy to mess up
• iterations have to go through entire graph
• Multi-Steps Workflow:
• each step in a separate Experiment
• uses saved datasets / readers / writers
• iterations come through individual steps separately

Machine Learning Templates from Azure Machine Learning Gallery

• Text classification (text tagging, text categorisation, etc). For example, assign piece of text to one or more pre-defined set of classes or categories:
• categorise articles
• organise web pages into hierarchical categories
• filter email spam
• search query user intend prediction
• support ticket team routing
• sentiment analysis
• feedback analysis
• Fraud detection
• Retail forecasting
• Predictive maintenance

Text Processing:

1. Language detection
2. Normalisation:
1. Convert words into normalised ones:
1. Down case: The -> the
2. Lemmatisation: plays -> play
3. Stemming: mice -> mouse
3. Stop-words removal: the, a, to, with, etc
4. Special character removal: for example, ignore all Non-Alpha-Numeric
5. Ignore numbers, emails, URLs, etc

Feature Extraction:

• 'Bags-of-words' - the document is treated as a set of words regardless order and grammar
• Split text into Bi-grams, tri-grams, n-grams
• Score how it correlates with your topic (e.g. sport, business, etc)
• Move from symbols to numbers:
• term occurrence (number of times words or n-grams appear in document)
• term frequency
• Inverse Document Frequency (IDF)
• the inverted rate of documents that contain words or n-grams against the whole training data set of documents
• downgrades the importance of useless words like 'which', 'the', etc
• TF-IDF
• frequent words that appear only in a small number of documents achieve high value
• Dimensionality Reduction (vocabulary used in documents could be too large) - 'curse of dimensionality'. To do:
• filter-based filter reduction
• wrapper-based filter reduction
• feature hashing
• topic modelling (LDA)

Model Training:

• Binary-class Learners:
• 2-class Logistic Regression
• 2-class Support Vector Machine
• 2-class Boosted Decision Tree
• Multi-class Learners:
• One-vs-All Muliclass
• Multi-class Logistic Regression
• Multi-class Decision Forest

Model Evaluation:

• Split data into Train, Development and Test subsets
• Compare and visualise the results of comparison between two trained models, e.g. N-grams and Uni-grams

Finally

Deploy your model on Azure as a web service. 

Easy!

:)

Content Enrichment

Content enrichment is about manipulating crawled content before it is added to the search index. For example, add a sentiment analysis score to indexed social activity.

Some components that support content enrichment are:

• Version control
• Technical metadata (formats, format versions, validation rules, etc)
• Provenance data (processing history)

Questions:

• How standardised is the enrichment information?
• How volatile is enriched information?
• When is the content enhanced (by author, during submission, during editorial, etc)?
• Where does enhanced information live (embedded, externally)?

Key challenges:

• What is the master source/copy of the information?
• Is the information normalised or de-normalised (repeating parent metadata across child elements)?
• How to synchronised across multiple systems?

NetKernel Bootcamp London - 17th August 2015

Speaking of the devil, 1060 Research has just announced about their bootcamp in London!

The event takes place on Monday 17th August and will be a full day introduction to Resource Oriented Computing (ROC) using NetKernel: ROC is what happens when you take micro-services to the limit and start to think in terms of resources as the first class concern of a software system.

The day will be hosted by 1060 Research at TechUK:

NetKernel Takes Micro-Services to the Ultimate Level

While talking about such relatively new boys on the market as Vert.x, Akka, Chronicle, Kafka, Ready! API, RxJava, etc which certainly are great components for solutions that respond to current demand for micro-services, the mainstream seems to be completely missing such nice, mature and easy to use product as NetKernel. The latter one is not competing with newcomers and together they can comprise quite elegant solutions that any architect would be eventually proud of.

NOTE: This is not a promotion for NetKernel. I don't work for them. This is just an attempt to be fair to those that somehow happened to be on a side of the road.

Functional Requirements for Commodity Price Engine

Introduction

Commodity Price Engine is a derivatives sales tool and potentially a trading application designed specifically for the commodities market covering energy, base metals and agricultural products. It provides server based pricing and sensitivities for structures consisting of forwards and options that incorporate volatility skew and is designed to be delivered via the web and as native mobile applica- tions. The implemented functionalities in the prototype are detailed below, along with market data requirements and planned extensions.

Supported Underlying Assets

Commodity Price Engine supports any asset with forward curves and implied volatility surfaces. This includes exchange traded products with sufficient liquidity and products for which the user is able to supply the forward curves and volatility surfaces. A planned extension for Commodity Price Engine would build required curves and surfaces to accommodate structures on illiquid underlying assets.

Supported Derivatives and Valuation

Pricing and sensitivities are available for forwards, bullet and Asian options, and structures consisting of any combination of forwards and options. The valuation model takes into account volatility skew and has been benchmarked against commercial software used in investment banks. Price and sensitivities can be converted to any currency and standard metric units.

4. Sales and Trading Features

Commodity Price Engine would allow addition of sales and trading margins, shifting of forward curves and volatility surfaces for what-if analysis, solving for break even strikes for structures, generation of term sheets, and graphing of forward curves and payoff diagrams. It also would accommodate back-dated pricing for available historical data.

5. Planned Extensions and Enhancements

Additional features that are planned for Commodity Price Engine include:
- Construction of illiquid forward curves and implied volatility surfaces.
- Calculation of credit value adjustment (CVA).
- Computation of value-at-risk (VaR).

6. Market Data Requirements

Commodity Price Engine assumes availability of the following market data:
- Yield curves for required currencies (it would be possible to bootstrap yield curves from cash, futures, OIS, swap, and single currency basis swap quotes).
- Forward curve and implied volatility surface (it is possible to build volatility surfaces from market quoted option prices) for required underlying assets.
- FX forward curve and volatility surface for required currency pairs.
- Implied survival probabilities for relevant entities if CVA calculation is required (it would be possible to compute the survival probabilities from yield curves and credit default swap (CDS) spread quotes).
- Historical data for above if VaR calculation is required.
Market data can be obtained from commercial data vendors such as Bloomberg or Reuters (commodities data from such market data sources as ze.com will need to be supplemented by interest rate, FX, and credit data).

7. Technology Architecture

Commodity Price Engine architecture consists of server and client side components. The server side manages market data and could be loosely coupled with a grid of quantitative pricing libraries. The client side is the Graphical User Interface (GUI) that communicates with the server via secure protocol and could be accessed from desktops or a variety of mobile devices. Pricing libraries could be placed on the client side if required.

8. Conclusion

Commodity Price Engine would be a sales and trading application designed for participants in the commodities market who traditionally relied on investment banks for pricing support due to limited access to suitable tools. It would have the capacity to become a full-scale trading platform if supplemented with modules for connecting to trade booking and counterparty portfolio management systems.

Microsoft Workshop: Developing for Internet of Things, London

The workshop took place in Microsoft office at 100 Victoria Street. The crowd was pretty big. First we came through a couple of presentations and then did three labs. Overall, the workshop was very interesting, it gave a good overview of what IoT consists of, where we are with it at the moment and how it would possibly evolve in near future. See below some take aways that I think could be helpful to review later on.

Our presenters were:

• Paul Foster, DX Microsoft UK, and
• Robert Hogg, MVP, Microsoft Integration, MD Black Marble

Some notes:

1. There are open source IoT frameworks (for example, check out AllJoyn)
2. IoT provides Data-Driven Insights (Telemetry):
1. More efficient use of resources (cost reduction, environmental impact)
2. More targeted products and services (social impact, increased revenue)
3. While working with connected devices, it's very hard to predict in advance what data will be useful. The important data may not be what was expected in the beginning. Therefore:
1. It's tempting but likely inefficient to try for business transformation in the first step.
2. Need to think about not only device telemetry but also diagnostic telemetry.
4. Privacy and security have to be addressed at very early stages.
5. Although the ability to control devices remotely could be quite helpful, in the beginning designers may need to get used to work with devices that provide one-way communication only.
6. Microsoft goal to support in Azure ANY device!
7. https://www.wirelessthings.net
8. Hortonworks Sandbox is a free installation of Hadoop that comes with sample data and tutorials. It could be installed on a personal computer - it's a great tool to start playing with real Hadoop.
9. Lots of interest in R programming. R is used in practically all universities across UK and investment banking. Many R scripts come for free from academia.
10. Practical Data Science and support for it is quite popular within nowadays business activity.
11. Microsoft provides free consulting advises for IoT initiatives.

Some slides:

1. It is expected that interest in IoT will get into initial peak then it may cool off with gradual and steady grows of popularity afterwards:

2. Different level of IoT evolution:

3. ToDo roadmap:

4. Variety of IoT devices:

5. IoT challenges:

6. Pattern to start with:

7. This is what Microsoft offers on Windows Azure for IoT:

8. Some IoT problems that could be solved with Windows Azure:

9.

10.

11. This Event Hub is already available in Windows Azure. In fact we used it in our first lab.

12. Stream Analytics is also already available in Windows Azure. We used it in out second lab.

13. Stream Analytics front-end in Windows Azure looks almost as simple as this diagram:

14. I'm not sure if it's really a 'pattern' but it's good to keep in mind that volume of incoming messages in IoT could be really huge:

15. Possible IoT participants:

16. This slide represents a great desire to keep IoThings under a tight control. We'll see if it would become a reality or stay just a dream:

17. This is what Event Hub on Windows Azure is capable of:

18. When I see such slides I think more and more about Lua, Barracuda Embedded Server and Express Logic:

19. Network security means encryption. I'm not quite sure why does a message from, say, a temperature sensor that has only two fields - IP address and temperature value - have to be encrypted? Keep in mind that millions of such messages would need to be decrypted at the Event Hub on arrival...

20. More about security:

21. It's good to know that there is the IoT Suite. We didn't play with it, so I don't really know how it looks like:

22. More concerns about IoT:

23. I guess that if you would follow one of the last two links, you might find this presentation in an original file:

24. These are three labs that I did on that day. First two required configuration on Windows Azure. In last one I used a Raspberry platform as a sensor that sends messages to the Event Hub configured in the first lab. I should admit that it was quite interesting to do this. Event Hub with Stream Analytics looked very similar to CEP (Complex Event Processing) that I worked with before.

25. Azure community is steadily growing. I have already booked a place for IoT & Data Hackathon in Reading and hope to put some info about it on the web as well:

26. It seems that topics on this slide and many more could be learnt on Microsoft workshops in London for free:

27. More events:

28. More links:

29. And more links:

A Potential Need for Commodity Price Engine

Introduction

Commodities market has experienced significant turbulence in recent times, possible returns and diversification benefits offered by commodities have attracted some investor interest. Derivatives have an important role to play in encouraging a further activity on this market, and this requires wide availability of pricing tools to improve price transparency and investor confidence. However, in contrast to other markets, there is an absence of such pricing tools for commodity derivatives due to their inherent complexities and this is the impetus behind the idea of development of a Commodity Price Engine described in this post.

Situation

Recent fluctuations in demand for raw materials is expected to continue for some time. High volatility in commodities market has led to certain growth in the derivatives market as commodity producers and consumers sought ways to hedge against adverse price movements.

When used for hedging purposes, futures contracts remove the risk of unexpected losses by providing price certainty, but for the same reason they also preclude the possibility of profiting from favourable price movements. As participants become more sophisticated, they naturally turn to options and other derivatives that allow them to obtain more flexible hedges and speculative positions.

At present, participants in the commodity derivatives market comprises primarily of large producers and consumers of raw materials, who have little choice but to use derivatives, usually over-the-counter (OTC), to hedge their positions, and large financial institutions that have the capacity to acquire necessary pricing tools to service this demand. But as regulators push more of these “standard” OTC derivatives onto exchanges to ensure greater transparency and competition, the derivatives market will attract broader class of investors attempting to take advantage of the benefits offered by commodities.

Complication

Although some commodity derivatives are already listed on exchanges and many others are traded over-the-counter, investors interested in entering this market are confronted with issues such as limited liquidity, poor quality of market data, and the absence of accurate pricing tools. These contribute towards the lack of transparency in the way commodity derivatives are valued, which adds to the perception of risks associated with these derivatives.

Liquidity and the quality of market data can only improve with greater activity in these derivatives, and for this to occur there must be more transparency and confidence in the way prices are determined.

Unfortunately, commodity derivatives have inherent complexities that require more advanced pricing tools than those used for derivatives in other markets. Although such tools do exist, their availability is limited to large financial institutions, and are included only in high-end commercial financial software. In order for the derivatives market to flourish, investors need a better understanding of the salient features of commodity derivatives and, more importantly, require access to quantitative tools for independent valuation of these derivatives with higher degree of confidence.

Solution

Commodity Price Engine could implement advanced pricing models for commodity derivatives and deliver these through platforms including the web, smartphones, and tablets. Salient properties of commodity derivatives and observed volatility skews in the market would be fully incorporated into the models to provide accurate valuation and flexible delivery platforms would ensure that these tools are available anywhere with access to the internet.

For reliability and scalability Commodity Price Engine could be deployed on a cloud computing infrastructure and be accompanied by a distributed data server that cleans and smoothes market data. The former ensures that intensive pricing calculations are available even on devices with limited computing power, while the latter eliminates, for most users, the non-trivial task of obtaining reliable market data.

In order to handle large number of concurrent user sessions, Commodity Price Engine could be enhanced with grid computing capabilities to ensure valuation requests receive faster responses even for complex derivatives and large portfolios. These features would enable small to medium sized market participants to independently value and monitor their derivative portfolios with confidence.

Conclusion

Higher returns and diversification benefits of commodities provide attractive trading opportunities and market participants seeking more tailored solutions for their requirements are naturally led to derivatives. With regulators pushing to move standard OTC derivatives onto exchanges, the demand for derivatives have a good chance to increase. A necessary catalyst to transform this increasing interest into growth in market activity is accessible quantitative tools that help bring transparency to this market, and this is precisely the role that Commodity Price Engine may play.