Wednesday, 26 August 2015

Content Enrichment

Content enrichment is about manipulating crawled content before it is added to the search index. For example, add a sentiment analysis score to indexed social activity.

Some components that support content enrichment are:

  • Version control
  • Technical metadata (formats, format versions, validation rules, etc)
  • Provenance data (processing history)

Questions:


  • How standardised is the enrichment information?
  • How volatile is enriched information?
  • When is the content enhanced (by author, during submission, during editorial, etc)?
  • Where does enhanced information live (embedded, externally)?

Key challenges:


  • What is the master source/copy of the information?
  • Is the information normalised or de-normalised (repeating parent metadata across child elements)?
  • How to synchronised across multiple systems?




Friday, 10 July 2015

NetKernel Bootcamp London - 17th August 2015

Speaking of the devil, 1060 Research has just announced about their bootcamp in London!

The event takes place on Monday 17th August and will be a full day introduction to Resource Oriented Computing (ROC) using NetKernel: ROC is what happens when you take micro-services to the limit and start to think in terms of resources as the first class concern of a software system.

The day will be hosted by 1060 Research at TechUK:


Thursday, 2 July 2015

NetKernel Takes Micro-Services to the Ultimate Level

While talking about such relatively new boys on the market as Vert.x, Akka, Chronicle, Kafka, Ready! API, RxJava, etc which certainly are great components for solutions that respond to current demand for micro-services, the mainstream seems to be completely missing such nice, mature and easy to use product as NetKernel. The latter one is not competing with newcomers and together they can comprise quite elegant solutions that any architect would be eventually proud of.

NOTE: This is not a promotion for NetKernel. I don't work for them. This is just an attempt to be fair to those that somehow happened to be on a side of the road.

Functional Requirements for Commodity Price Engine

Introduction

Commodity Price Engine is a derivatives sales tool and potentially a trading application designed specifically for the commodities market covering energy, base metals and agricultural products. It provides server based pricing and sensitivities for structures consisting of forwards and options that incorporate volatility skew and is designed to be delivered via the web and as native mobile applica- tions. The implemented functionalities in the prototype are detailed below, along with market data requirements and planned extensions.

Supported Underlying Assets

Commodity Price Engine supports any asset with forward curves and implied volatility surfaces. This includes exchange traded products with sufficient liquidity and products for which the user is able to supply the forward curves and volatility surfaces. A planned extension for Commodity Price Engine would build required curves and surfaces to accommodate structures on illiquid underlying assets.

Supported Derivatives and Valuation

Pricing and sensitivities are available for forwards, bullet and Asian options, and structures consisting of any combination of forwards and options. The valuation model takes into account volatility skew and has been benchmarked against commercial software used in investment banks. Price and sensitivities can be converted to any currency and standard metric units.

4. Sales and Trading Features

Commodity Price Engine would allow addition of sales and trading margins, shifting of forward curves and volatility surfaces for what-if analysis, solving for break even strikes for structures, generation of term sheets, and graphing of forward curves and payoff diagrams. It also would accommodate back-dated pricing for available historical data.

5. Planned Extensions and Enhancements

Additional features that are planned for Commodity Price Engine include:
- Construction of illiquid forward curves and implied volatility surfaces.
- Calculation of credit value adjustment (CVA).
- Computation of value-at-risk (VaR).

6. Market Data Requirements

Commodity Price Engine assumes availability of the following market data:
- Yield curves for required currencies (it would be possible to bootstrap yield curves from cash, futures, OIS, swap, and single currency basis swap quotes).
- Forward curve and implied volatility surface (it is possible to build volatility surfaces from market quoted option prices) for required underlying assets.
- FX forward curve and volatility surface for required currency pairs.
- Implied survival probabilities for relevant entities if CVA calculation is required (it would be possible to compute the survival probabilities from yield curves and credit default swap (CDS) spread quotes).
- Historical data for above if VaR calculation is required.
Market data can be obtained from commercial data vendors such as Bloomberg or Reuters (commodities data from such market data sources as ze.com will need to be supplemented by interest rate, FX, and credit data).

7. Technology Architecture

Commodity Price Engine architecture consists of server and client side components. The server side manages market data and could be loosely coupled with a grid of quantitative pricing libraries. The client side is the Graphical User Interface (GUI) that communicates with the server via secure protocol and could be accessed from desktops or a variety of mobile devices. Pricing libraries could be placed on the client side if required.

8. Conclusion

Commodity Price Engine would be a sales and trading application designed for participants in the commodities market who traditionally relied on investment banks for pricing support due to limited access to suitable tools. It would have the capacity to become a full-scale trading platform if supplemented with modules for connecting to trade booking and counterparty portfolio management systems.

Tuesday, 19 May 2015

Microsoft Workshop: Developing for Internet of Things, London

The workshop took place in Microsoft office at 100 Victoria Street. The crowd was pretty big. First we came through a couple of presentations and then did three labs. Overall, the workshop was very interesting, it gave a good overview of what IoT consists of, where we are with it at the moment and how it would possibly evolve in near future. See below some take aways that I think could be helpful to review later on.

Our presenters were:

  • Paul Foster, DX Microsoft UK, and
  • Robert Hogg, MVP, Microsoft Integration, MD Black Marble

Some notes:
  1. There are open source IoT frameworks (for example, check out AllJoyn)
  2. IoT provides Data-Driven Insights (Telemetry):
    1. More efficient use of resources (cost reduction, environmental impact)
    2. More targeted products and services (social impact, increased revenue)
  3. While working with connected devices, it's very hard to predict in advance what data will be useful. The important data may not be what was expected in the beginning. Therefore:
    1. It's tempting but likely inefficient to try for business transformation in the first step.
    2. Need to think about not only device telemetry but also diagnostic telemetry.
  4. Privacy and security have to be addressed at very early stages.
  5. Although the ability to control devices remotely could be quite helpful, in the beginning designers may need to get used to work with devices that provide one-way communication only.
  6. Microsoft goal to support in Azure ANY device!
  7. https://www.wirelessthings.net
  8. Hortonworks Sandbox is a free installation of Hadoop that comes with sample data and tutorials. It could be installed on a personal computer - it's a great tool to start playing with real Hadoop.
  9. Lots of interest in R programming. R is used in practically all universities across UK and investment banking. Many R scripts come for free from academia.
  10. Practical Data Science and support for it is quite popular within nowadays business activity.
  11. Microsoft provides free consulting advises for IoT initiatives.

Some slides:

1. It is expected that interest in IoT will get into initial peak then it may cool off with gradual and steady grows of popularity afterwards:


2. Different level of IoT evolution:


3. ToDo roadmap:


4. Variety of IoT devices:


5. IoT challenges:


6. Pattern to start with:

7. This is what Microsoft offers on Windows Azure for IoT:

8. Some IoT problems that could be solved with Windows Azure:

9.
10.
11. This Event Hub is already available in Windows Azure. In fact we used it in our first lab.

12. Stream Analytics is also already available in Windows Azure. We used it in out second lab.

13. Stream Analytics front-end in Windows Azure looks almost as simple as this diagram:

14. I'm not sure if it's really a 'pattern' but it's good to keep in mind that volume of incoming messages in IoT could be really huge:

15. Possible IoT participants:

16. This slide represents a great desire to keep IoThings under a tight control. We'll see if it would become a reality or stay just a dream:

17. This is what Event Hub on Windows Azure is capable of:

18. When I see such slides I think more and more about Lua, Barracuda Embedded Server and Express Logic:

19. Network security means encryption. I'm not quite sure why does a message from, say, a temperature sensor that has only two fields - IP address and temperature value - have to be encrypted? Keep in mind that millions of such messages would need to be decrypted at the Event Hub on arrival...

20. More about security:

21. It's good to know that there is the IoT Suite. We didn't play with it, so I don't really know how it looks like:

22. More concerns about IoT:

23. I guess that if you would follow one of the last two links, you might find this presentation in an original file:

24. These are three labs that I did on that day. First two required configuration on Windows Azure. In last one I used a Raspberry platform as a sensor that sends messages to the Event Hub configured in the first lab. I should admit that it was quite interesting to do this. Event Hub with Stream Analytics looked very similar to CEP (Complex Event Processing) that I worked with before.

25. Azure community is steadily growing. I have already booked a place for IoT & Data Hackathon in Reading and hope to put some info about it on the web as well:

26. It seems that topics on this slide and many more could be learnt on Microsoft workshops in London for free:

27. More events:

28. More links:

29. And more links:




Friday, 1 May 2015

A Potential Need for Commodity Price Engine

Introduction

Commodities market has experienced significant turbulence in recent times, possible returns and diversification benefits offered by commodities have attracted some investor interest. Derivatives have an important role to play in encouraging a further activity on this market, and this requires wide availability of pricing tools to improve price transparency and investor confidence. However, in contrast to other markets, there is an absence of such pricing tools for commodity derivatives due to their inherent complexities and this is the impetus behind the idea of development of a Commodity Price Engine described in this post.

Situation

Recent fluctuations in demand for raw materials is expected to continue for some time. High volatility in commodities market has led to certain growth in the derivatives market as commodity producers and consumers sought ways to hedge against adverse price movements.

When used for hedging purposes, futures contracts remove the risk of unexpected losses by providing price certainty, but for the same reason they also preclude the possibility of profiting from favourable price movements. As participants become more sophisticated, they naturally turn to options and other derivatives that allow them to obtain more flexible hedges and speculative positions.

At present, participants in the commodity derivatives market comprises primarily of large producers and consumers of raw materials, who have little choice but to use derivatives, usually over-the-counter (OTC), to hedge their positions, and large financial institutions that have the capacity to acquire necessary pricing tools to service this demand. But as regulators push more of these “standard” OTC derivatives onto exchanges to ensure greater transparency and competition, the derivatives market will attract broader class of investors attempting to take advantage of the benefits offered by commodities.

Complication

Although some commodity derivatives are already listed on exchanges and many others are traded over-the-counter, investors interested in entering this market are confronted with issues such as limited liquidity, poor quality of market data, and the absence of accurate pricing tools. These contribute towards the lack of transparency in the way commodity derivatives are valued, which adds to the perception of risks associated with these derivatives.

Liquidity and the quality of market data can only improve with greater activity in these derivatives, and for this to occur there must be more transparency and confidence in the way prices are determined.

Unfortunately, commodity derivatives have inherent complexities that require more advanced pricing tools than those used for derivatives in other markets. Although such tools do exist, their availability is limited to large financial institutions, and are included only in high-end commercial financial software. In order for the derivatives market to flourish, investors need a better understanding of the salient features of commodity derivatives and, more importantly, require access to quantitative tools for independent valuation of these derivatives with higher degree of confidence.

Solution

Commodity Price Engine could implement advanced pricing models for commodity derivatives and deliver these through platforms including the web, smartphones, and tablets. Salient properties of commodity derivatives and observed volatility skews in the market would be fully incorporated into the models to provide accurate valuation and flexible delivery platforms would ensure that these tools are available anywhere with access to the internet.

For reliability and scalability Commodity Price Engine could be deployed on a cloud computing infrastructure and be accompanied by a distributed data server that cleans and smoothes market data. The former ensures that intensive pricing calculations are available even on devices with limited computing power, while the latter eliminates, for most users, the non-trivial task of obtaining reliable market data.

In order to handle large number of concurrent user sessions, Commodity Price Engine could be enhanced with grid computing capabilities to ensure valuation requests receive faster responses even for complex derivatives and large portfolios. These features would enable small to medium sized market participants to independently value and monitor their derivative portfolios with confidence.

Conclusion

Higher returns and diversification benefits of commodities provide attractive trading opportunities and market participants seeking more tailored solutions for their requirements are naturally led to derivatives. With regulators pushing to move standard OTC derivatives onto exchanges, the demand for derivatives have a good chance to increase. A necessary catalyst to transform this increasing interest into growth in market activity is accessible quantitative tools that help bring transparency to this market, and this is precisely the role that Commodity Price Engine may play.

Monday, 27 April 2015

Global Azure Bootcamp, London

When I found out about the bootcamp from Meetup announcement, it was too late to get a seats there as all of them were booked out in no time at all. Luckily, just a couple of days before it, I could get my pass from someone who changed his plans for that Saturday.

The meeting took place in New Zealand House, just steps away from Trafalgar Square. The office had a creative look-n-feel and had enough space for all participants. Morning tea and lunch were nice and quite filling. Speed of wifi connection and visibility of slides projected on a big white wall were just excellent.

Unfortunately, we didn't do as many labs as it was planned in the beginning of the meeting but nevertheless presentations were informative and comments, based on personal experience of the presenters (Richard Conway and Tiberiu Covaci), were very interesting and valuable.

Below there are some notes that I took there:

  1. Microsoft started offering Nano Server - a minimal footprint installation of Windows Server that is highly optimized for the cloud, and ideal for containers. Basically, it's a bare minimum operating system good (and fast!) enough to run C#, Java, Python, Node.js and PHP applications on the cloud.
  2. Azure REST API is clearly versioned when Amazon's is not. This means that if you run your application on Azure, you could stick with a certain version of API and avoid any surprises after its possible upgrade. On the other hand, upgrades on AWS may come out of the blue and most likely affect (usually negatively) your application.
  3. It was mentioned that Visual Studio Online (VSO) is not exactly a proper environment for a serious product development. In fact, VSO is a TFS Online with Git integration (users could choose either Github or Bitbucket). For those who love TFS, VSO could be useful if they embark on Agile (Scum) development.
  4. There were two interesting comments about Microsoft Azure Marketplace:
    1. This is a good place to publish your server-based app and start selling it to the world, and
    2. Current workflow for publishing your app there is somewhat cumbersome. The whole process is described in a 700+ pages manual and it may take up to few months to move your app over there. Good news is that Microsoft would like to compress this process to two weeks, bad news is that at the moment it's not clear when exactly that compression would happen.
  5. Deployment to Azure could be completely automated. JSON (deployment) file may contain a complete script for creating the whole infrastructure (boxes, VPNs, connections, etc, etc).
  6. Microsoft product range include IaaS, PaaS and SaaS. Interestingly, it was noted that majority of people still think that Azure services are too expensive. Well, I thought the same. According to my personal experience last year one box on Azure cost me around $80/month and I could run exactly the same set of apps on DigitalOcean for $20/month when I migrated there. I guess, I should check Azure again (Nano Server!), maybe it did become cheaper since then.
  7. On Azure application services could perform Web Roles and Worker Roles. Web Roles come with IIS7 (or IIS8) and different versions of .NET libraries. Worker Roles could be responsible for queues polling, event listening, external process management, etc. Basically, a system that requires extensive parallel processing (like CVA calculations on a compute grid) could be compiled using such roles without the need of multi-threaded programming. Roles are defined in a Hosted Service. At runtime each Role will execute on one or more instances.
  8. Integrated development experience powered by Visual Studio and Azure SDK includes .NET, Java, Node.js, Python and PHP.
  9. Deploying on a Cloud requires a different mindset:
    1. It comes with unusual (at first) errors, and
    2. Design for availability, reliability and scalability would differ from ones done for apps that run on local machines.
  10. Azure scaling consists of:
    1. Up scaling - choosing different VM sizes
    2. Out scaling - adding more instances
    3. Auto-scaling - built-in functionality in application blocks
    4. Out scaling - by using multiple service entities
    5. Caching to offset server workloads
  11. Data Management on Azure
    1. There various data storage engines available:
      1. SQL Server
      2. DocumentDB (NoSQL)
      3. Redis
      4. Etc, etc.
    2. Available plans:
      1. Basic
      2. Standard, and
      3. Premium
    3. Non-distruptive replication is provided
    4. Data retention in Azure cloud backup is for 7, 14 and 35 days
    5. Data storages could be accessed via REST API, PowerShell or Azure Portal
    6. Azure SQL Server doesn't provide:
      1. Profiler
      2. Native Encryption
      3. SQL Agent
      4. CLR
      5. Service Broker
      6. Distributed transactions and
      7. Distributed Views
    7. Azure SQL Database Management Portal is a web access for Azure SQL Server.
    8. There are readily available images for a range of standard SQL products, such as SQL Sever, Oracle, MySQL, etc that could be quickly and easily deployed on a cloud VM. This combines the power of cloud VMs with full features of SQL engines. It's good for enterprises (multiple DBs on the same box for different environments, etc) but not really needed for eCommerce websites. Ideally, one needs to compare SQL database vs SQL IaaS, there are (+)s and (-)s for different solution designs (research more about FullText Search, Windows Authentication for VMs joined to on-premises domain, large databases, etc).
    9. There is a fully managed, scalable JSON document database service. It took 18 months for Microsoft to release this long awaited service!
    10. DocumentDB has certain advantages over MongoDB:
      1. MongoDB is not not easy to scaleable for an existing apps.
      2. There are problems with migration of MongoDB (I'm not sure what use cases it is for).
    11. Machine Learning. Although on Azure it was originally meant to be used on a single VM, soon it would be possible to scale it out with help of Revolutionary Analytics. I really liked this part and would definitely try it as soon as come across a proper task for it. The features are:
      1. No coding required.
      2. It's possible to use your own or open source analytic libraries (for example R or Python).
      3. Briefly the workflow steps look as follows:
        1. Get connected to a data source (some database with tables);
        2. Select columns that you are interested in;
        3. Define input parameters;
        4. Build your path (you can call it a Model);
        5. Train your Model (feed it with as much historical data as possible);
        6. Evaluate the Model (85% is way better than 30%);
        7. Deploy your Model as a web service with just one click.
  12. Microsoft Azure Customer Connection Program (CCP) could be useful for those who would really like to get more intimate connections with this cloud (ECGCC@microsoft.com)
Overall I liked this meeting and look forward to get onto similar ones that might take place in London in near future.



Online Encyclopedia of Statistical Science (Free)

Please, click on the chart below to go to the source: