Microsoft Workshop: Developing for Internet of Things, London

The workshop took place in Microsoft office at 100 Victoria Street. The crowd was pretty big. First we came through a couple of presentations and then did three labs. Overall, the workshop was very interesting, it gave a good overview of what IoT consists of, where we are with it at the moment and how it would possibly evolve in near future. See below some take aways that I think could be helpful to review later on.

Our presenters were:

• Paul Foster, DX Microsoft UK, and
• Robert Hogg, MVP, Microsoft Integration, MD Black Marble

Some notes:

1. There are open source IoT frameworks (for example, check out AllJoyn)
2. IoT provides Data-Driven Insights (Telemetry):
1. More efficient use of resources (cost reduction, environmental impact)
2. More targeted products and services (social impact, increased revenue)
3. While working with connected devices, it's very hard to predict in advance what data will be useful. The important data may not be what was expected in the beginning. Therefore:
1. It's tempting but likely inefficient to try for business transformation in the first step.
2. Need to think about not only device telemetry but also diagnostic telemetry.
4. Privacy and security have to be addressed at very early stages.
5. Although the ability to control devices remotely could be quite helpful, in the beginning designers may need to get used to work with devices that provide one-way communication only.
6. Microsoft goal to support in Azure ANY device!
7. https://www.wirelessthings.net
8. Hortonworks Sandbox is a free installation of Hadoop that comes with sample data and tutorials. It could be installed on a personal computer - it's a great tool to start playing with real Hadoop.
9. Lots of interest in R programming. R is used in practically all universities across UK and investment banking. Many R scripts come for free from academia.
10. Practical Data Science and support for it is quite popular within nowadays business activity.
11. Microsoft provides free consulting advises for IoT initiatives.

Some slides:

1. It is expected that interest in IoT will get into initial peak then it may cool off with gradual and steady grows of popularity afterwards:

2. Different level of IoT evolution:

3. ToDo roadmap:

4. Variety of IoT devices:

5. IoT challenges:

6. Pattern to start with:

7. This is what Microsoft offers on Windows Azure for IoT:

8. Some IoT problems that could be solved with Windows Azure:

9.

10.

11. This Event Hub is already available in Windows Azure. In fact we used it in our first lab.

12. Stream Analytics is also already available in Windows Azure. We used it in out second lab.

13. Stream Analytics front-end in Windows Azure looks almost as simple as this diagram:

14. I'm not sure if it's really a 'pattern' but it's good to keep in mind that volume of incoming messages in IoT could be really huge:

15. Possible IoT participants:

16. This slide represents a great desire to keep IoThings under a tight control. We'll see if it would become a reality or stay just a dream:

17. This is what Event Hub on Windows Azure is capable of:

18. When I see such slides I think more and more about Lua, Barracuda Embedded Server and Express Logic:

19. Network security means encryption. I'm not quite sure why does a message from, say, a temperature sensor that has only two fields - IP address and temperature value - have to be encrypted? Keep in mind that millions of such messages would need to be decrypted at the Event Hub on arrival...

20. More about security:

21. It's good to know that there is the IoT Suite. We didn't play with it, so I don't really know how it looks like:

22. More concerns about IoT:

23. I guess that if you would follow one of the last two links, you might find this presentation in an original file:

24. These are three labs that I did on that day. First two required configuration on Windows Azure. In last one I used a Raspberry platform as a sensor that sends messages to the Event Hub configured in the first lab. I should admit that it was quite interesting to do this. Event Hub with Stream Analytics looked very similar to CEP (Complex Event Processing) that I worked with before.

25. Azure community is steadily growing. I have already booked a place for IoT & Data Hackathon in Reading and hope to put some info about it on the web as well:

26. It seems that topics on this slide and many more could be learnt on Microsoft workshops in London for free:

27. More events:

28. More links:

29. And more links:

A Potential Need for Commodity Price Engine

Introduction

Commodities market has experienced significant turbulence in recent times, possible returns and diversification benefits offered by commodities have attracted some investor interest. Derivatives have an important role to play in encouraging a further activity on this market, and this requires wide availability of pricing tools to improve price transparency and investor confidence. However, in contrast to other markets, there is an absence of such pricing tools for commodity derivatives due to their inherent complexities and this is the impetus behind the idea of development of a Commodity Price Engine described in this post.

Situation

Recent fluctuations in demand for raw materials is expected to continue for some time. High volatility in commodities market has led to certain growth in the derivatives market as commodity producers and consumers sought ways to hedge against adverse price movements.

When used for hedging purposes, futures contracts remove the risk of unexpected losses by providing price certainty, but for the same reason they also preclude the possibility of profiting from favourable price movements. As participants become more sophisticated, they naturally turn to options and other derivatives that allow them to obtain more flexible hedges and speculative positions.

At present, participants in the commodity derivatives market comprises primarily of large producers and consumers of raw materials, who have little choice but to use derivatives, usually over-the-counter (OTC), to hedge their positions, and large financial institutions that have the capacity to acquire necessary pricing tools to service this demand. But as regulators push more of these “standard” OTC derivatives onto exchanges to ensure greater transparency and competition, the derivatives market will attract broader class of investors attempting to take advantage of the benefits offered by commodities.

Complication

Although some commodity derivatives are already listed on exchanges and many others are traded over-the-counter, investors interested in entering this market are confronted with issues such as limited liquidity, poor quality of market data, and the absence of accurate pricing tools. These contribute towards the lack of transparency in the way commodity derivatives are valued, which adds to the perception of risks associated with these derivatives.

Liquidity and the quality of market data can only improve with greater activity in these derivatives, and for this to occur there must be more transparency and confidence in the way prices are determined.

Unfortunately, commodity derivatives have inherent complexities that require more advanced pricing tools than those used for derivatives in other markets. Although such tools do exist, their availability is limited to large financial institutions, and are included only in high-end commercial financial software. In order for the derivatives market to flourish, investors need a better understanding of the salient features of commodity derivatives and, more importantly, require access to quantitative tools for independent valuation of these derivatives with higher degree of confidence.

Solution

Commodity Price Engine could implement advanced pricing models for commodity derivatives and deliver these through platforms including the web, smartphones, and tablets. Salient properties of commodity derivatives and observed volatility skews in the market would be fully incorporated into the models to provide accurate valuation and flexible delivery platforms would ensure that these tools are available anywhere with access to the internet.

For reliability and scalability Commodity Price Engine could be deployed on a cloud computing infrastructure and be accompanied by a distributed data server that cleans and smoothes market data. The former ensures that intensive pricing calculations are available even on devices with limited computing power, while the latter eliminates, for most users, the non-trivial task of obtaining reliable market data.

In order to handle large number of concurrent user sessions, Commodity Price Engine could be enhanced with grid computing capabilities to ensure valuation requests receive faster responses even for complex derivatives and large portfolios. These features would enable small to medium sized market participants to independently value and monitor their derivative portfolios with confidence.

Conclusion

Higher returns and diversification benefits of commodities provide attractive trading opportunities and market participants seeking more tailored solutions for their requirements are naturally led to derivatives. With regulators pushing to move standard OTC derivatives onto exchanges, the demand for derivatives have a good chance to increase. A necessary catalyst to transform this increasing interest into growth in market activity is accessible quantitative tools that help bring transparency to this market, and this is precisely the role that Commodity Price Engine may play.

Global Azure Bootcamp, London

When I found out about the bootcamp from Meetup announcement, it was too late to get a seats there as all of them were booked out in no time at all. Luckily, just a couple of days before it, I could get my pass from someone who changed his plans for that Saturday.

The meeting took place in New Zealand House, just steps away from Trafalgar Square. The office had a creative look-n-feel and had enough space for all participants. Morning tea and lunch were nice and quite filling. Speed of wifi connection and visibility of slides projected on a big white wall were just excellent.

Unfortunately, we didn't do as many labs as it was planned in the beginning of the meeting but nevertheless presentations were informative and comments, based on personal experience of the presenters (Richard Conway and Tiberiu Covaci), were very interesting and valuable.

Below there are some notes that I took there:

1. Microsoft started offering Nano Server - a minimal footprint installation of Windows Server that is highly optimized for the cloud, and ideal for containers. Basically, it's a bare minimum operating system good (and fast!) enough to run C#, Java, Python, Node.js and PHP applications on the cloud.
2. Azure REST API is clearly versioned when Amazon's is not. This means that if you run your application on Azure, you could stick with a certain version of API and avoid any surprises after its possible upgrade. On the other hand, upgrades on AWS may come out of the blue and most likely affect (usually negatively) your application.
3. It was mentioned that Visual Studio Online (VSO) is not exactly a proper environment for a serious product development. In fact, VSO is a TFS Online with Git integration (users could choose either Github or Bitbucket). For those who love TFS, VSO could be useful if they embark on Agile (Scum) development.
4. There were two interesting comments about Microsoft Azure Marketplace:
1. This is a good place to publish your server-based app and start selling it to the world, and
2. Current workflow for publishing your app there is somewhat cumbersome. The whole process is described in a 700+ pages manual and it may take up to few months to move your app over there. Good news is that Microsoft would like to compress this process to two weeks, bad news is that at the moment it's not clear when exactly that compression would happen.
5. Deployment to Azure could be completely automated. JSON (deployment) file may contain a complete script for creating the whole infrastructure (boxes, VPNs, connections, etc, etc).
6. Microsoft product range include IaaS, PaaS and SaaS. Interestingly, it was noted that majority of people still think that Azure services are too expensive. Well, I thought the same. According to my personal experience last year one box on Azure cost me around $80/month and I could run exactly the same set of apps on DigitalOcean for $20/month when I migrated there. I guess, I should check Azure again (Nano Server!), maybe it did become cheaper since then.
7. On Azure application services could perform Web Roles and Worker Roles. Web Roles come with IIS7 (or IIS8) and different versions of .NET libraries. Worker Roles could be responsible for queues polling, event listening, external process management, etc. Basically, a system that requires extensive parallel processing (like CVA calculations on a compute grid) could be compiled using such roles without the need of multi-threaded programming. Roles are defined in a Hosted Service. At runtime each Role will execute on one or more instances.
8. Integrated development experience powered by Visual Studio and Azure SDK includes .NET, Java, Node.js, Python and PHP.
9. Deploying on a Cloud requires a different mindset:
1. It comes with unusual (at first) errors, and
2. Design for availability, reliability and scalability would differ from ones done for apps that run on local machines.
10. Azure scaling consists of:
1. Up scaling - choosing different VM sizes
2. Out scaling - adding more instances
3. Auto-scaling - built-in functionality in application blocks
4. Out scaling - by using multiple service entities
5. Caching to offset server workloads
11. Data Management on Azure
1. There various data storage engines available:
1. SQL Server
2. DocumentDB (NoSQL)
3. Redis
4. Etc, etc.
2. Available plans:
1. Basic
2. Standard, and
3. Premium
3. Non-distruptive replication is provided
4. Data retention in Azure cloud backup is for 7, 14 and 35 days
5. Data storages could be accessed via REST API, PowerShell or Azure Portal
6. Azure SQL Server doesn't provide:
1. Profiler
2. Native Encryption
3. SQL Agent
4. CLR
5. Service Broker
6. Distributed transactions and
7. Distributed Views
7. Azure SQL Database Management Portal is a web access for Azure SQL Server.
8. There are readily available images for a range of standard SQL products, such as SQL Sever, Oracle, MySQL, etc that could be quickly and easily deployed on a cloud VM. This combines the power of cloud VMs with full features of SQL engines. It's good for enterprises (multiple DBs on the same box for different environments, etc) but not really needed for eCommerce websites. Ideally, one needs to compare SQL database vs SQL IaaS, there are (+)s and (-)s for different solution designs (research more about FullText Search, Windows Authentication for VMs joined to on-premises domain, large databases, etc).
9. There is a fully managed, scalable JSON document database service. It took 18 months for Microsoft to release this long awaited service!
10. DocumentDB has certain advantages over MongoDB:
1. MongoDB is not not easy to scaleable for an existing apps.
2. There are problems with migration of MongoDB (I'm not sure what use cases it is for).
11. Machine Learning. Although on Azure it was originally meant to be used on a single VM, soon it would be possible to scale it out with help of Revolutionary Analytics. I really liked this part and would definitely try it as soon as come across a proper task for it. The features are:
1. No coding required.
2. It's possible to use your own or open source analytic libraries (for example R or Python).
3. Briefly the workflow steps look as follows:
1. Get connected to a data source (some database with tables);
2. Select columns that you are interested in;
3. Define input parameters;
4. Build your path (you can call it a Model);
5. Train your Model (feed it with as much historical data as possible);
6. Evaluate the Model (85% is way better than 30%);
7. Deploy your Model as a web service with just one click.
12. Microsoft Azure Customer Connection Program (CCP) could be useful for those who would really like to get more intimate connections with this cloud (ECGCC@microsoft.com)

Overall I liked this meeting and look forward to get onto similar ones that might take place in London in near future.

An Idea for a Telco Company

1. CURRENT SITUATION

Introduction of the Cloud removed any constrains in scaling infrastructures for SME companies. Nowadays PaaS and SaaS kind of services became a reality and in many cases it’s a practical and only way to extend business operations without substantial investments in expensive hard- and software.

The nature of software packages evolved to accommodate new runtime environments. Instead of keeping heavy software packages on individual platforms, their compute power is moved to horizontally scalable Cloud servers and their functionality is handled remotely via nice and intuitively understandable GUIs easily accessible from any stationary or mobile device. Modernised and newly created products from Microsoft, Adobe and Google are good examples.

Current technology trends are not industrially finalised yet and keep changing traditional approaches. For the purpose of this message, we could highlight three current distinctive practices:

-    Vendors deliver functionality in a modularised manner, when a consumer can pick up only useful services and pay accordingly,

-    Modularised functionality could be easily extended via well defined APIs that are progressively becoming standard part of such services, and

-    Deployment of customised environments with all selected modules (or services) could be up to 100% automated.

In regards to technical support and maintenance, consumer market has changed as well. Customers try to avoid unnecessary expenses for keeping in-house IT departments and started looking for available services (or service providers) elsewhere. Such demand created a new market for taking services from one or more vendors, combining them into bundles and selling them to right buyers.

2. IDEA

Given that, on the one hand, the market is getting saturated with new cloud-based modularised soft- and hard-ware services empowered by standardised APIs and, on the other hand, consumers look for customisable solutions, it could be a perfect time to build a business that connects one with another.

Such ‘connection’ could be done by creation of reusable scripts (or programs) that would automatically deploy a whole environment for an individual or a group of customers. Such environments should include build-in plug-ins for handling customer-to-Telco and Telco-to-vendor billing.

Scripts could be archived in a Library that would naturally grow covering more and more use cases. The Library would become company ‘know-how’ and its value could be measured by the number of use cases it handles.

3. POSSIBLE SOLUTION

a.     Allocate Cloud servers.

b.    Find vendors of modularised and API-enabled PaaSes and SaaSes.

c.     Create a team to build scripts for automated deployments.

d.     Organise development from epicentre out in agile manner covering more and more functionality within each cycle:

a.     Collect consumer requirements and find most popular scenarios (use cases).

b.    Build and test scripts to meet business requirements in current cycle.

c.     Deploy new environments and allocate some resources for their initial maintenance.

d.     Move scripts to the Library and repeat (d).

REFERENCES:

• APIs for Dummies
• Chief
• Puppet
• Salt Stack
• Docker
• Ansible
• Amazon EC2 API

OTN ADF Mobile Workshop @ Oracle Offices in Singapore

From marketing perspective this workshop gave a pretty good overview for what Oracle is doing in regards to concurring mobile platforms with Java. From technical perspective, the exercises we've done there were rather simple and could've been better ones - it would be more fun to build something with database connections and more sophisticated browsing.

Anyway, this is what was discussed about Oracle JDeveloper and its ADF Mobile Extension (ADF stands for Application Development Framework):

SOME MARKETING STUFF:

1. ADF Mobile components have sufficient number of widgets to build a decent application for iOS and Android from single code base.

2. Oracle implementation of mobile security seems to be the most shining component in the whole stack. Sorry, I can't say more than that because presentation of security features didn't really happen due to some technical issues.

3. Those who have experience working with Oracle ADF objects, could start doing things with Mobile ADF in no time at all.

4. Currently, Mobile ADF components can be used for generating application deployment packages only for iOS and Android. Windows 8 is not in the list. Not yet. Few years ago Oracle got ready for Windows 6 that had never come out! Now they want to wait and see if Windows 8 spills out of marketing campaigns and acquire some real popularity. If Windows 8 gets its market share, it would be relatively easy for Oracle, as they say, to accommodate it within their Mobile ADF.

5. GUI is done in Javascript, HTML5 and CSS3, so it's compatible with any current browser and mobile device. This is so called Hybrid Application when this kind of GUI works on top of a Java container that nicely communicates with whatever hardware is below. Platform-specific JVM is an old Java trick and in this case it's coupled with a widely used GUI technology.

SOME TECHNICAL STUFF:

1. Size of a final application ready to be deployed on a mobile OS would be around 10Mb+ because it includes headless Java Virtual Machine. JVM was cut off all GUI classes hence 'headless'. Basically, it's based on JavaME/JDK1.4 and has no Java5 features like collections and annotations.

2. Oracle mobile application would communicate with native OS and hardware resources (like email, camera, GPRS) via PhoneGap layer. Well, now it's PhoneGap but Oracle will switch it to Cordova because PhoneGap is owned by Adobe and there are some license related issues with that.

3. Mobile ADF Components are free only for developing prototypes. Any plans for using them within commercial products have to be discussed with Oracle sales people.

CONCERNS:

1. Every single mobile app built with Mobile ADF would come with its own JVM... Is that good? Or it doesn't matter? I'm not quite sure... Mobile operating systems are multi-threaded now. Some apps may run in the background tracking your location, popping up messages, making alerts. In a year time, an average phone might have, say, 25 apps constantly running in the background. Would it be good (size-wise, efficiency-wise) if some of them contain their own run-time environments? It's like to run few Eclipse IDEs at once on a desktop, isn't it?

2. Price is always the issue. Small businesses may not like it. Big businesses may stick to their existing development platforms, whatever they currently use.

ADVANTAGES:

Well, there are four actually:

1. Generation of hybrid-native deployment packages from a single code base. It does sound a bit scary, especially for those companies that already have separate iOS and Android development teams. Will that 'single code base' really work? Hmm... it's hard so say... but there is always an alternative - just start hiring Windows 8 team now.

2. Maybe it's true and Oracle mobile security solutions really stand out on the market (this needs to be confirmed...).

3. Millions of Java developers can apply their skills in mobile sector, and

4. The whole thing is backed by Oracle and it's not about to disappear from this market. What does that mean? Well, Cordova is open source - its future is not quite certain. PhoneGap belongs to a bit unpredictable Adobe - people still remember what they have done with Flex.


Probably it's time to check what ApplicationCraft is...

Microsoft Office on Linux in 2014?

What if it's true and after all we'll get a full version of Microsoft Office available on Linux platform? Let's see who are the main players in this space:

DESKTOP

1. OpenOffice.org (free)

2. LibreOffice (free)

3. SoftMaker Office ($79.95)

4. IBM Lotus Symphony (free, based on OpenOffice)


5. GNOME Office (free)


6. KOffice (free)


7. StarOffice (free)



8. Calligra (free)

9. Yozo Office ($59)

10. Breadbox ($99)

ONLINE

- Zoho Office ($3/user)

- ThinkFree Online Office ($49)
- Google Docs Office Suite (free)
- Microsoft Office 365 ($4-20 per user/month)
- CollateBox ($9.99-129.99/month for unlimited number of users)

It's pretty clear that Microsoft won't be alone out there. It would be quite interesting to see how competition would change the lists above.

Open Source Financial Trading Software

Some time in early 2012 an open source project Lodestone Foundation was backed by Deutsche Bank. In September, 2012 FT let it know to ones who missed the news earlier.

The initiative is quite interesting and it split finance professionals in two camps - ones who believe that this is the way to go and their counterparts who think it would never happen.

I've looked for any existing and more or less active projects in this space. This is what I found out so far:

1. OpenGamma (Java, R, .Net and REST APIs, Tools for MS Excel)

The OpenGamma Platform provides both buy-side and sell-side firms a rich set of features, including trade data management, live risk alerts, and sophisticated, dynamic Excel integration.

2. QuantLib (C++)

QuantLib project is aimed at providing a comprehensive software framework for quantitative finance. QuantLib is a free/open-source library for modeling, trading, and risk management in real-life. 

Possibly this is the most popular project at the moment, there are signs that it's quietly used by many quant developers in their day-to-day work.

3. Open Heartbeat (probably C/C++)

Open Heartbeat is a lightweight data distribution system designed primarily for collecting and distributing rapidly changing price quotes for stocks, securities or other distributable data sets.

The core component is a reusable library which manages all communication, synchronization  caching etc., which can be linked into applications that create or, more commonly, consume data.

Primarily aimed at the financial sector, as a way to distribute rapidly changing stock quotes etc., this system can be used for any application that needs to distribute volatile data sets reliably and efficiently.

This suite includes a sample feed handler for Yahoo financial data. Use of the data provided by this service is subject to the terms and conditions of the Yahoo real time stock quote service.

The library offers very efficient operation (~10,000 updates per second), redundant paths and automatic error recovery and reporting.

4. Eclipse Trader (Java)

EclipseTrader is an  Eclipse Rich Client Platform (RCP) application focused on the building of an online stock trading system, featuring shares pricing watch, intraday and history charts with technical analysis indicators, level II/market depth view, news watching, and integrated trading. 

The standard Eclipse RCP plug-ins architecture allows third-party vendors to extend the functionality of the program to include custom indicators, views or access to subscription-based data feeds and order entry.

5. QuickFX/J (Java)

The Financial Information eXchange (FIX) protocol is a messaging standard developed specifically for the real-time electronic exchange of securities transactions. FIX is a public-domain specification owned and maintained by FIX Protocol, Ltd (FPL).

QuickFIX/J is a fully featured messaging engine for the FIX protocol. It is a 100% Java open source implementation of the popular C++ QuickFIX engine.

6. Premium Markets (Java)

Premium Markets is an automated financial technical analysis system. 
It implements a graphical environment for monitoring financial technical analysis major indicators and for portfolio management.

In its advanced packaging (not provided under open-source license) it also includes :

• Screening of financial web sites to pickup the best market shares, 
• Forecast of share prices trend changes on the basis of financial technical analysis, (with a rate of around 70% of forecasts being successful observed while back testing over DJI, FTSE, DAX and SBF), 
• Back testing and Email sending on buy and sell alerts triggered while scanning markets and user defined portfolios.

7. ojAlgo (Java)

ojAlgo is Open Source Java code that has to do with mathematics, linear algebra and optimization  Its feature set make it particularly suitable for use within the financial domain.

8. EWaveTrade (C++, MFC, Visual C++ Compiler)

EWaveTrade is a tool to aid in the process of analyzing market price data using the Elliot Wave Theory, particularly applying the concepts and principles developed by Glenn Neely. The book "Mastering Elliott Wave" by Glen Neely is recommended for both potential developers and users of this software.

9. QChartist (Qt, Basic)

QChartist is a free charting software designed to do technical analysis from any data. The program is written in Basic language.

10. jFin (Java)

Pure Java lib that provides:

• Date adjustment
• Schedule generation
• Day count fraction calculation

11. ActiveQuant (Java, ActiveMQ, HSQL)

ActiveQuant is an entire framework to backtest and run automated trading systems. This project is an open source project, but is not a free software, it is provided free of charge to academics and hobbyists.

12. Akutan (Java)

Akutan is an open source finance project working in the asset allocation, portfolio analysis arena of financial informatics. For the most part it stays away from instrument valuation and market models given that there are several other projects dedicated to that space, e.g. quantlib.

13. Net Positions

Net Positions allows to receive current and historical data about net positions(*) and opened on the futures market (info is taken from 'Commitments of Traders' weekly report). 

(*) net position is the difference between total open long and open shot positions.

14. Credit Analytics

CreditAnalytics is a full-featured financial fixed income credit analytics, trading, and risk library, developed with a special focus towards the needs of the credit products community. In particular, CreditAnalytics provides analytics to value liquid products (CDS, CDX, CDO, and bonds of all types and variants) and standard index/custom products (single credit forwards and options, and portfolio credit forwards, options, tranches, and other structures).

15. FpML

FpML® (Financial products Markup Language) is the open source XML standard for electronic dealing and processing of OTC derivatives. It establishes the industry protocol for sharing information on, and dealing in, financial derivatives and structured products.

The standard is developed under the auspices of ISDA, using the ISDA derivatives documentation as the basis. As a true open standard, the standards work is available to all at no cost and open to contribution from all. There is no membership requirement.

The standard evolution and development is overseen and managed by the FpML Standards committee, following W3C rules of operations guidelines.

16. Kooderive

The objective of this project is to create a library of code for pricing financial derivatives products using CUDA to achieve GPU programming.