architecture, cloud

Azure cli provides powerful tools before Portal is finally polished

But first , the tricky part is using Azure’s CLI tool . I would suggest using node version:

Installing : npm -g install azure-cli

Once you’re done, they require you to upload .publishsettings file , oddly named its also not easily accessible. Run the following command

azure account download

Above command will actually open a browser and if you’re logged in (azure) you will be able to download it.

Next step is importing it

azure account import filename.publishsettings

 

At this point you want to switch to Resource Manager mode, its a newer API architecture where everything is configurable via JSON structure and this is inline with new Portal management. The following command will switch you to that mode:

azure config mode arm

Few more handy resources when dealing with azure-cli

Using the Azure CLI for Mac, Linux, and Windows with Azure Service Management

Share This:

architecture, coding

Open sourcing my lottery server

2007 project involving Message Driven Beans (EJB) running on Enterprise Java container (Jboss or Glassfish) where games are processed and “played” as they come in through the Queue allowing scalability and consistency.

 

Source code for both game logic and EJB server can be found on github:

Github to Lottery Server

Share This:

architecture, bigdata, mongodb

MongoDB 3.0 worth the upgrade

MongoDB 3.0 (dev version of 2.8) has been almost a year in the making and there are several prominent changes that are worth mentioning:

 

  1. Document level locking – changes to those large json documents (aka rows ) will work similarly to regular RDBMS. Previously you had to either queue or create a custom way of handling inconsistencies.
  2. Document has a hard limit of 16megs (reasonable) and that allows your app to create massive amount of data that will require sharding your servers. A new storage engine WiredTiger will provide better compression for storage, allegedly up to 80% , shed those Gigs of data Foursquare!
  3. Besides other significant improvements, a new tool is introduced named Ops Manager, uncertain if it would be available to all or MMS subscribers.
  4. Looking into MongoDB Jira , there is a large set of bugs that have been fixed, better error handling in GEO lookups, syncing issues between nodes and others.

 

 

A few notes:

 

In order to take advantage of a new pluggable storage engine wiredTiger you need to explicitly configure it as such:

 

mongod --storageEngine wiredTiger

Overall great job!

 

 

Share This:

architecture, javascript, nodejs

Official nodejs.org is dead?, long live IOJS

It appears that IOJS (io.js, a form of node.js ) has gained an incredible momentum and credibility. As a seasoned nodeJS engineer i would be strongly swayed towards iojs give that it has all the important committee members (committers to nodejs).

My recent observation of this fork is that they release every 4 days ! That’s an impressive frequency.

Although some are uncertain about this fork in hopes that official (rather original) nodejs may all of a sudden resolve their differences and release 0.12 or perhaps even Nodejs 1.0, we shall see what happens in the next few months.
At one point a plan was to release NodeJS 0.12 in March 2014 , prediction is that it could be this year.

Meanwhile , Strongloop is gaining ground as a promenant NodeJS consulting firm with an array of tools (from monitoring , web ide generating rest apis, etc).

 

To add to this note, folks are realizing that NPM INSTALL may be something devops have missed and insecure (or plain old malware) modules can be installed ( might be even root).

 

 

Feb 11 UPDATE: Official NodeJS 0.12 was released just few days ago albeit without mention of generators. Great news!

 

Share This:

architecture, bigdata, coding

Processing data using your GPU cores – with or without OpenCL

I’ve been personally fascinated by the progress big data computing is making these days. Few weeks i’ve been experiment with h2o (in memory java cluster wide math) that processes all kinds of algorithms across multiple clusters, ALL in memory.

What eluded me to understand is what’s happening with using those GPUs we have in our everyday laptops and desktops.

 

I’ve recently reached out to a lead engineer for AMD’s Aparapi project and asked him what’s happening with the project, after all there hasn’t been a release in over a year!

Gary Frost – lead engineer and contributor to Aparapi wrote:

It is active, but mostly in the lambda branch.   if you like AMD GPU's (well APU's) you will love the lambda branch it allows you to use Java 8 lambda features *AND* allows you to execute code on HSA enabled devices (no OpenCL required).  This means no buffer transfers and much less restrictions on the type of code you can execute. 

So the new API's map/match the new Java 8 stream APIs

Aparapi.range(0,100).parallel().forEach(id -> square[i]=in[i]*in[i]);

If you drop the 'parallel' the lambda is executed sequentially, if you include the parallel Aparapi looks for HSA, then OpenCL then if neither exist will fall back to a thread pool. 

The reason that there are less 'checkins' in trunk, and no merges from lambda into trunk is because we can;t check the lambda stuff into trunk without forcing all users to move to Java 8. 

This is really exciting and interesting !

In a more practical use case, what i would image doing is running map reduce with data provided (e.g. hazecast or other datagrid) for each node and utilize AMD’s finest GPUs to process data much quicker than costly Xeon 12 cores can do.This provides an affordable scalability.

For the time being, there will still be a need to have Hadoop’s data/name nodes and job tracker that would control which piece of data is process and where since GPUs won’t be able to share data between their remote nodes (at least for now).

Next steps to try it out:

Check out branch “lambda”

and Compile/Run

Aparapi.range(0,100).parallel().forEach(id -> square[i]=in[i]*in[i]);

There are plenty of other examples in the project’s source.

 

Enjoy

 

Share This:

architecture, mongodb

Why i still love MongoDB

After reading a Slashdot article on MongoDB vs DynamoDB (chosen) i’d like to summarize why DynamoDB became less attractive to me:

 

  • Architect/Developer chose to serialize a collection of objects into JSON and save that field as one column – this screams future problems in searching and simple lookups
  • Apparently (this may have changed) there is 64Kb limit on the item , some people start to compress data (json)
  • Manually rebuilding indexes – no comment here
  • Pricing model has to be predefined – number of reads per second, although affordable vs dedicated cloud based mongodb deployment (given that i am not sure if performance numbers are the same)
  • No way to embed objects , you’d have to create infinite amount of columns (similar to cassandra)

 

 

Share This:

architecture, javascript

Publishing to AWS SNS topic

SNS is wonderful , supporting HTTP, email(No HTML markup), email-json, SQS and SMS (US only for now).

Basically it allows you to receive various types of notifications or an app subscribed via various protocols.

Let’s bypass discussion how to subscribe to a Topic , its pretty trivial and is very flexible (some thirdparty can receive an HTTP call based on a message that arrived, awesome, right? )

Here is an example using NodeJS and AWS node library of publishing a message.

var AWS = require('aws-sdk');
//don't hard code your credential 🙂
 AWS.config.update({accessKeyId: 'ZZZ', secretAccessKey: 'ZZZ', region: 'us-east-1'});

var sns = new AWS.SNS();
sns.publish(
{TopicArn:'arn:aws:sns:us-east-1:283fdfdf-warning', Message:"Just testing for now ", Subject: "testing "}
, function(err,data){
 if (err){
     log("Error sending a message "+err);
 }else{
     log("Sent message {0}".format(data.MessageID));
 }
 });

 

Share This:

architecture, coding, ruby

Simple threading/async execution in Ruby – tip on rake

Ever need to suspend a task or a small job in Ruby (rails included) ? Of course you do. I won’t get into the lengthy discussion about green-threads in Ruby and yes Ruby is single threaded, JRuby can and does use  JVMs java.lang.Thread , in fact there is a very handy safe class  you should use “ScheduledThreadPoolExecutor”.

 

Celluloid

I love the existence of this very very handy gem, it covers most of the situations for you to “call and forget” when it comes to asynchronous tasks. It is based on the concept of Actors and Watchers(Observers). In fact it reminds me of Java’s AOP but it’s not it.

This gem is super easy to use.

  1. Obviously add it to gemfile
  2. Add “include Celluloid” to your class and …
  3. call “async.<method of that class” and

Celluloid will handle the rest, it’ll figure out if it needs to use fiber, fork or java’s thread.

You can observe and control your actors or just leave it.

Here is the tip, sometimes we need to execute rake task (say  scheduled task) , you would assume that celluloid will work just fine. it WON’T.

What happens is that rake will shutdown the process and cellulo”id” method will never be execute AND you won’t get an error.

How to make it work?

simple, just add “require ‘celluloid/autostart'” to your rake file and everything will be asynchronously run.

 

Thanks & hope it helps

 

 

Share This:

architecture, coding, javascript, mongodb

Skedul.In project is wonderful – power of google API and ruby

Not always do you get to work with wonderful client that know what they want, but i have. Skedul.In soon to be launched in production mode (currently in beta) is a simple yet great idea: have single place to create all your events at Once , select your google calendar, use your google contacts and you are done, all events are created, invitation sent, your calendar updated.

 

Techie stuff:

Ruby 2.0

Rails 3.2.X

MongoDB – Can’t live without it , also used for session storage

Completely relies on OpenAuth , you have to login with your Google Account

Google Calendar API, Contacts API

Hosted on Heroku

 

Couple of screenshots :

Screenshots

Share This:

architecture

How Postgres works behind the scene

We all like pictures and what a better way to show processing of a query then this flowchart:

postgresqlflow

Share This: