Rethinking Enterprise Apps

Enterprise IT departments often run 100s, if not 1000s of applications. The scope of products deployed is vast and complex. Enterprise software is accumulated over many years and is typically a combination of in-house developed, third party or packaged as a service running in the cloud (or someone else’s data centre).

Most IT engineers have a story of maintaining a server that runs in the darkest corner of the network. No one is sure who owns it, what it does or wants to make a change because when it breaks, you’re in for a rough few days!

Managing Enterprise workloads is a considerable undertaking. It takes time, money and discipline to build an IT function which enables the business and supports the user base. Companies continuously grapple with the issue; how can you empower the business while keeping things under control? How do we provide value while dealing with old technologies and antiquated systems?

In my recent post, I suggested blockchain technology is an option for revolutionising Enterprise infrastructure, driving meaningful transformation. How does this relate to Enterprise, how do blockchain platforms lower complexity, build trust, integrity and improve the corporate operating experience?

It is my belief, open-source Smart Contract platforms provide a comprehensive base platform for enterprise applications. The technology distributes processing and storage geographically and solves many of our infrastructure challenges of today. Furthermore, integrity and security are fundamental components and are a system default.

I consider a Smart Contract platform as a new form of Serverless technology. The use case is similar; you execute your contract (code) on infrastructure, which is owned and operated by someone else. You pay for the service with Gas on an Ethereum based chains or by holding and staking utility tokens on EOSIO (note, that’s a subject for another post). The importance of hardware should not be understated; however, as an IT Engineer, it is not your direct responsibility. Your job is to minimise the Gas spent, lower the CPU execution time and limit the bandwidth consumed. The goal is to focus on what’s critical to the business, which are customer-facing or business-enhancing applications which create value.

Take, for example, the open-source software EOSIO. It operates on bare metal servers, cloud instances, containers or a standard laptop (primarily for development). A blockchain is Internet native, meaning latency, jitter, link failures and infrastructure problems won’t impact a well-architected platform. You are not beholden to a single vendor and could utilise any suitable infrastructure, owned or rented. The code you build on top is built-in modules, inside smart contracts and can be used by any person, app or machine with the appropriate permissions.

If Enterprise were to starting rewriting apps on a blockchain, many of the things we do manually today would happen automatically. Real-world events are a trigger for actions. In the system, all the items are tracked, audited and is immutable. This paradigm allows us to boil down an app to what’s essential and beneficial.

The question is, why can’t we use existing technologies? Why introduce blockchain when centralised applications and databases have worked for years? For me, the answer is simple. Blockchains present an open format which promotes sharing of information, irrespective of geography, distance, network links, hardware, vendors and clouds. The functions of a Smart Contract may be transparent, well defined and available to anyone with permissions. There are many reasons to consider the technology which I’ll look to expand on in a future post.

If you’ve come this far, thanks for sticking around!! Feel free to connect if you’d like to discuss.

Restarting Prism – Version 4.1.x and Later

One of the most popular posts on my blog is “Nutanix: Restarting Prism”, the Nutanix user interface which I wrote way back 2015. Three years is a long time in technology, especially in the hyper-converged world so I thought it was time for a quick update.

The following will enable you to restart prism services on nodes running AOS 4.1.x or later

Determine the current Prism Cluster Leader

Connect via ssh to any AOS host in your cluster and run:

ssh admin@{any_node_ipaddress}
Nutanix Controller VM
admin@{any_node_ipaddress}'s password: (default: nutanix/4u)
curl http://0:2019/prism/leader && echo


{"leader":"{node_leader_ipaddress}:9080", "is_local":true}

{node_leader_ipaddress} is the active cluster leader for the prism service. If you’re connected to the leader AOS host, “is_local” will be true. In this case, we need to ssh to another AOS host to restart the service.

Stop the Prism Service on the Leader

Connect via ssh to the prism leader AOS host and run:

ssh admin@{node_leader_ipaddress}
Nutanix Controller VM
admin@{node_leader_ipaddress}'s password: (default: nutanix/4u)

genesis stop prism


2018-09-09 18:46:55.797151: Stopping prism (pids [6579, 6607, 6608, 6645, 23894, 23933])</strong>
2018-09-09 18:46:56.380999: Services running on this node:
  insights_data_transfer: [6140, 6237, 6238, 6264, 6266, 6267, 6268]
  cluster_health: [2016, 2017, 2108, 2109, 2111, 2112, 2114, 2115, 2119, 2120, 2122, 2123, 2134, 2152, 7234, 7259, 7260, 7938, 8013, 8014, 10985, 10986]
  nutanix_guest_tools: [6980, 7025, 7026, 7038]
  pithos: [5816, 5877, 5878, 5936]
  cerebro: [6167, 6307, 6308, 6487]
  delphi: [7924, 7979, 7980, 7981]
  aplos_engine: [7745, 7805, 7806, 7807]
  uhura: [6728, 6859, 6860, 6861]
  acropolis: [6686, 6806, 6807, 6809]
  cluster_config: [7721, 7778, 7779, 7780]
  alert_manager: [6614, 6676, 6677, 6762]
  stargate: [6131, 6207, 6208, 6370, 6371]
  foundation: []
  curator: [6252, 6353, 6354, 6443]
  genesis: [2231, 2252, 2275, 2276, 3733, 3735]
  lazan: [7894, 7970, 7971, 7973]
  insights_server: [6134, 6197, 6198, 6321]
  minerva_cvm: [7717, 7749, 7750, 7752, 7888]
  snmp_manager: [6775, 6890, 6891, 6892]
  mantle: [5820, 5904, 5905, 5952]
  catalog: [6678, 6798, 6799, 6800]
  hera: [5825, 5928, 5929]
  chronos: [6211, 6325, 6326, 6372]
  sys_stat_collector: [6813, 6928, 6929, 6931]
  secure_file_sync: [5332, 5388, 5389, 5390]
  ergon: [6146, 6258, 6259, 6261]
  arithmos: [6619, 6718, 6719, 6893]
  dynamic_ring_changer: [5812, 5871, 5872, 5946]
  prism: []
  zookeeper: [2483, 2513, 2514, 2520, 2575, 2592]
  aplos: [7890, 7920, 7921, 7922, 8058, 8060]
  scavenger: [3611, 3640, 3641, 3642]
  ssl_terminator: [5328, 5360, 5361, 5362]
  janus: [6862, 6967, 6968]
  cim_service: [6610, 6666, 6667, 6683]
  tunnel_manager: [6832, 6959, 6960]
  cassandra: [2038, 5461, 5625, 5655, 5656]

All other services stay running, there should be no disruption to cluster storage services.

Start the Prism Service on the Leader

Stay connected to the prism leader AOS host and run:

cluster start
[sudo] password for admin:


2018-09-09 18:47:42 INFO cluster is attempting to connect to Zookeeper
2018-09-09 18:47:42 INFO cluster:2609 Executing action start on SVMs {node_1},{node_2},{node_3}
Waiting on {node_1} (Up) to start:
Waiting on {node_2} (Up) to start:
Waiting on {node_3} (Up, ZeusLeader) to start:  Prism

Waiting on {node_1} (Up) to start:
Waiting on {node_2} (Up) to start:
Waiting on {node_3} (Up, ZeusLeader) to start:  Prism

Waiting on {node_1} (Up) to start:
Waiting on {node_2} (Up) to start:
Waiting on {node_3} (Up, ZeusLeader) to start:  Prism

Waiting on {node_1} (Up) to start:
Waiting on {node_2} (Up) to start:
Waiting on {node_3} (Up, ZeusLeader) to start:  Prism

Waiting on {node_1} (Up) to start:
Waiting on {node_2} (Up) to start:
Waiting on {node_3} (Up, ZeusLeader) to start:  Prism

Waiting on {node_1} (Up) to start:
Waiting on {node_2} (Up) to start:
Waiting on {node_3} (Up, ZeusLeader) to start:  Prism

Waiting on {node_1} (Up) to start:
Waiting on {node_2} (Up) to start:
Waiting on {node_3} (Up, ZeusLeader) to start:  Prism

Waiting on {node_1} (Up) to start:
Waiting on {node_2} (Up) to start:
Waiting on {node_3} (Up, ZeusLeader) to start:  Prism

Waiting on {node_1} (Up) to start:
Waiting on {node_2} (Up) to start:
Waiting on {node_3} (Up, ZeusLeader) to start:  Prism

Waiting on {node_1} (Up) to start:
Waiting on {node_2} (Up) to start:
Waiting on {node_3} (Up, ZeusLeader) to start:  Prism

Waiting on {node_1} (Up) to start:
Waiting on {node_2} (Up) to start:
Waiting on {node_3} (Up, ZeusLeader) to start:  Prism

Waiting on {node_1} (Up) to start:
Waiting on {node_2} (Up) to start:
Waiting on {node_3} (Up, ZeusLeader) to start:  Prism

Waiting on {node_1} (Up) to start:
Waiting on {node_2} (Up) to start:
Waiting on {node_3} (Up, ZeusLeader) to start:  Prism

Waiting on {node_1} (Up) to start:
Waiting on {node_2} (Up) to start:
Waiting on {node_3} (Up, ZeusLeader) to start:  Prism

Waiting on {node_1} (Up) to start:
Waiting on {node_2} (Up) to start:
Waiting on {node_3} (Up, ZeusLeader) to start:  Prism

Waiting on {node_1} (Up) to start:
Waiting on {node_2} (Up) to start:
Waiting on {node_3} (Up, ZeusLeader) to start:

Prism should now be operational. Refresh the browser.

Hope this helps! Until next time.


Disabling the Password

Over the past month, I’ve been holidaying with my family in Germany and Denmark. My wife, Britta, comes from a small farming community in the north of Germany near Hamburg and every couple of years we drop in to visit her family and friends.

Image result for map germany and denmark

I’ve travelled to Germany six times in the last twelve years and each time I offer to clean up her parent’s computer, do updates, install patches and this trip, added additional memory to their iMac. I find it’s the simplest way to keep the Skype communication channels open for weekly calls and serves as a thank you for allowing us to live in their house and eat their food!

It’s fair to say my in-laws are not the most computer savvy and have little interest in using the computer for anything other than Skype. The mac gets used once or twice a week at the absolute most, it’s sole purpose is to be a video phone so by the time we visit, updates are long overdue.

Over the last 12 years, each time I’ve performed an upgrade or installed a new version of Apple OS, we battled with locked accounts or forgotten passwords. This got me thinking… has nothing changed in the last decade? Managing passwords is still tricky, time-consuming and insecure, especially for your day to day end user!

I’m sure most people in technology have experienced this phenomenon when helping out family or friends. You go to your parents place thinking there’ll be an easy fix, only to spend the next two days resetting an account and proving their identity. Attackers have improved their capabilities in the last ten years, stealing passwords with relative ease, taking peoples accounts and credit card numbers. Head to your favourite IT news site, and you’ll find a new article about the latest breach, it’s almost a daily occurrence.

Image result for hacking funny security

I started thinking about my situation. What do I do, what’s my process? It’s certainly not perfect but here it is anyway.

  1. I use a password database and set random passwords for all sites I visit. I rotate my primary password a few times per year.
  2. Use Multi-Factor Authentication for everything that supports it. Easy to set up (in most cases), difficult when buying a new phone.
  3. I store a backup of my database in an encrypted file offline. Just in case, you never know when a software service might stop working or disappear.
  4. I keep database backup codes in a safe location; you can never be too sure
  5. Commit the most critical passwords to memory and never store them digitally

I’m sure you have similar strategies, but I think everyone can agree, managing and storing passwords is a lot of work, and if you’re like my in-laws you don’t have the knowledge or interest. If you’re an enterprise, you spend substantial amounts of money on this problem every day, and it’s a considerable cost to the business.

So is there an answer to the problem? Or are we stuck with passwords forever? In my previous post, I suggested that blockchain and the cryptographic technologies may be used to render passwords obsolete. Numerous startups are looking to solve this problem, for example, Civic and Remme, just to name a couple.

The question here is why use Blockchain or a decentralised system over something centralised like a certificate authority? Certainly, digital certificates can negate the need for passwords; however, it’s still a requirement to trust the authority. What if it gets compromised or sells your credentials? If they are large enough, they will be a target for attack, look at the companies and governments that trusted Equifax with their users’ data.

I like the idea of using a decentralised blockchain solution for just that reason. A single organisation does not control it, compromising a node doesn’t compromise the entire network, and if you control the keys, you control the account.

Anyone who has used or even played with blockchain technology such as Ethereum (ETH) or EOSIO would note the current difficulty and learning curve associated with the technology. I agree, key pairs are not intuitive to use, however, just like email or the browser was foreign in 1996, usability improves over time, allowing anyone to get involved.

Take the EOSIO chain as an example, the network launched in June 2018 with a CLI wallet, no simple user interface or block explorers. In around three months, the ecosystem has developed, the community stepped in to build some incredible solutions for people with low technical literacy or care factor. The pace of change is accelerating and will only keep improving, eventually making the system user-friendly enough for the mainstream.

Today’s blockchain technologies use cryptographic keys which the user must understand and maintain themselves. In the future, blockchains will provide user-friendly account names, automatic key creation and storage with integration into a smartphone. Users will not realise they are using a Blockchain, just as most don’t understand anything about the base layer protocols of the Internet. My in-laws do not know TCP/IP or RTSP. This knowledge is not required to use Skype or WhatsApp. They call by clicking a picture and answer when it rings.

A raft of new applications or “dApps” is currently in development, using the blockchain at the base protocol layer. If you’re an ETH account holder, you can interact with the dApps built on ETH via tools such as MetaMask, a browser plugin. The same is true for EOSIO, create your account then interact with the dApps using Scatter and your account name. For those in Enterprise familiar with single sign-on, the concept is similar, except you don’t own the user. The user comes to your dApp and consumes the product, or in the case of EOSIO you can assist to onboard them to the network, but you never see their new private key as it’s generated by their local device.

At no point is a password exchanged between the user and the dApp. The user interacts by sending transactions, which are messages signed by the users private key by an app such as Scatter or MetaMask. The network is aware of the users public key so can easily distinguish between real or fraudulent transactions, it can then determine if a transaction is valid based on the network rules and it’s consensus mechanism. The security is built into the protocol and is not an afterthought or an add-on! In the case of EOSIO, there are additional options around permissions, time delay and staking available to improve the overall security and enable features such as account recovery.

I look forward to the day where my password database is reduced to handful of keys, secured by a hardware device, most likely my phone with faceID or another form of physical biometric security. The time is approaching, and I hope it comes quick! After all these years, we need to make the password obsolete.

Thanks for reading!

In future posts, I plan to dive into some of the services that are operating on the blockchain and demonstrate how Scatter can be used to access multiple products with a single account.

Until next time.

Blockchain and Crypto

I’m a big fan of cryptocurrencies and blockchain technology. Even though I’d heard about Bitcoin in the early days, I didn’t appreciate or understand the implications of the technology and what it will mean for business in the next two decades. In early 2017, a friend reintroduced me to crypto, and since then I’ve been hooked on the possibilities.

In my day job, I’m a technology strategist. I evaluate new tech and work with senior leaders and executives to integrate, deploy and support new products and services. Over the last 5-6 years, I can honestly say I’ve been relatively successful at picking winning solutions.

For example, I was an early adopter of the Nutanix HCI platform and worked tirelessly to implement the product in my workplace of the day. Our deployment was mostly successful, sure, there were some teething issues, but overall things were smooth and delivered business benefits. Since then the on-premises storage and computing market has exploded with HCI solutions. Major vendors such as HPE, Dell, Cisco and Lenovo all have all moved with the market to deliver HCI products to the masses.

In early 2015 I started investigating and deploying SDWAN technologies. For those who don’t know, SDWAN is a software-defined solution that helps a network engineer control their traffic flows between data centres, branch offices and the cloud using any carriage. SDWAN solutions are designed with smarts, enabling the user to buy Internet bandwidth and run their WAN networks across the public Internet without the hassle of manual and time consuming configuration. SDWAN helps to break the stranglehold of the MPLS carrier on the enterprise. There are now several choices available to the network architect, and if you understand the business requirement, you can engineer a suitable cost-effective solution that provides more bandwidth at a significantly reduced cost. SDWAN is becoming the default WAN solution, most customers now demand these features when assessing solutions, and it’s reshaping the industry.

That brings me back around to cryptocurrencies and blockchain. I believe the technology has the traits and characteristics of a truly transformational technology which will change how we build applications and as a result, will dramatically improve and simplify infrastructure delivery.

So what are these properties? What makes blockchain and crypto transformational for the infrastructure layer? Why do I think it will be useful for the enterprise? I’ll briefly cover some topics here, by no means is the list exhaustive, and in future posts, I plan to expand into the detail.

Hybrid IT

For years the vision of enterprise IT has been the hybrid cloud. Take a look around the industry and count the vendors building a hybrid solution. For anyone in Infrastructure, the thought of choosing the appropriate location for your workload based on characteristics such as cost, performance, resilience etc. has been a long standing goal. Unfortunately, it hasn’t materialised! I believe blockchain with Inter Blockchain Communication (IBC) will one day solve this challenge. As an example, imagine the option to hold public user information on an open chain (usernames, DOB, keys etc.), but interact with private data on a closed chain running in your enterprise network. We may finally see true hybrid IT in action! Picture your business interacting with existing users on a blockchain; you don’t control their accounts, key or information; you use it with their permission, lowering your overall risk and exposure to hacking and data loss.

Standardised Infrastructure

New Blockchains like EOSIO run on standards x86 servers and use consensus protocols that don’t require traditional mining. In my opinion, this is fabulous for the enterprise, as most enterprises desire a simplified infrastructure, and are not prepared to increase costs through the purchase of specialised ASCIs or GPU mining products. Enterprise wants standard building blocks on which to build and deploy applications. Using building blocks addresses data centre complexity and decreased the management overhead. Software such as EOSIO can run anywhere, on-prem using bare metal, cloud, or on a platform as a service solution and is deployed using modern tools and automation. There is much more to discuss in this space, including why I think using a decentralised blockchain is a winner over central databases, but let’s save that topic for another day.

Retiring the Password

Blockchains uses public/private key pairs and cryptography to trust and authenticate users. Coupling blockchain with the secure storage of keys and access via biometrics or technology like Apple’s FaceID will change how enterprise users connect to their business environments. Anyone in tech knows the perils of password management and security threats that relate to weak user passwords or mismanaged passwords. I’m looking forward to the day when the password is obsolete.

Rethinking the Database

Enterprise and business spend substantial amounts of hard earned cash on database systems and vendor support every year! It’s the cost of doing business, right? What if you could redirect that money into a standard application delivery platform that provides distributed storage for your most essential data on a pay for use model? Use the benefits of the open source community to help you achieve your business goals while contributing back to the community. It’s possible in the blockchain and open source community, and in the future, I’ll expand on some of the possibilities.

Network Security

I believe blockchain has the potential to help enterprise and business eliminate the highly centralised network security model. If you’ve worked for any reasonably sized company, you would no doubt have waited anywhere from days to weeks, maybe even months to get a firewall rule to allow someone to connect to your service. Moving to a system that natively builds in security and trust, moves the security to where it should be, in the application. Doing away with passwords as discussed above helps us move the security perimeter from the edge firewall to the edge application. I’m excited about the possibilities here and is something I will cover in future posts.

Decentralising the Internet

There is lots of debate around just how centralised the Internet has become (check out the great article on Medium by @vincetabora). As a long-term goal for blockchain, many believe it will help to redistribute traffic away from the facebook, amazon and googles of the world. Being a fan of blockchain and crypto I hope it will help us here, but I can’t see this change taking place quickly.

Wrap Up

If you got this deep into my post, thanks for reading! I guess some may find it strange an IT guy with an Infrastructure background is writing and singing the praises of blockchain and crypto. After observing the problems for so long, I fundamentally believe this technology will help transform enterprise infrastructure, no matter where it lives, on-premises or cloud.

Until next time!

Nutanix – Redeploying nodes with Foundation


I recently deployed a Nutanix cluster which was a combination of old and new 3000 series nodes. Setup of the new machines went according to plan, however I found the old servers were reporting incorrect block IDs and/or positions.

For those who aren’t familiar with 3000 series systems, the product is Nutanix badged SuperMicro hardware, 4 servers in 2U of rack space. Each chassis has 4 slots (A-D).

Photo: 3000 series nodes


To fix the issue you need to update the factor settings on the Controller VM (CVM). Here are the steps:

  1. Document your node locations, double and triple check.
  2. Step two varies based on your hypervisor. In my case I was configuring an ESXi cluster so I needed to Foundation before I could expand the cluster.
  3. Run a scan in Foundation. Cross reference this with your doco from step 1.
  4. If it doesn’t match, edit the “factory_config.json” file which is found in “/etc/nutanix/” directory on each node. This assumes you’ve already configured your IPMI or have direct access to the physical equipment.
  5. Update the “rackable_unit_serial:”, “node_position:” or both to match your layout. Caution! Json files are particular… Don’t screw up the formatting.
  6. Restart the Genesis service by typing “genesis restart” at the $ prompt.
  7. Re-run Foundation. Nodes should now correspond to the desired layout.

Good luck.

Why is it so: UCCX Chrome Support (or lack of)

I recently worked on a UCCX 9 to 10 upgrade. First thing I tried after finalising the upgrade wasn’t the new features, it was browser support! I remember saying to my colleague, “surely they support it now!” But to my surprise, I clicked into UCCX from Chrome and found the same issue with the drop down menus! No change, no improvement, no freakin’ way!

I find it hard to believe Cisco does not support Google Chrome in Cisco Unified Contact Centre Express (UCCX). It works in other products. The lack of consistency is inexcusable…

According to Forbes, Chrome now has over 20% of marketshare which exceeds Firefox, which is of course supported. Come on, how hard would it be to fix the menus! It would take almost no time, it’s probably a 5 minute change to improve user experience…

So I raised a TAC Case, after all we pay big bucks to get support, only to get given the same old “we don’t support it, speak to your account team”.

What a joke! Please Cisco get this sorted…

Use Case Driven Design

I think it’s fair to say the majority of people working in networking have had some experience with Cisco, after all, Cisco is still the industry leader in a $50B dollar market. When I first switched to networking, I knew nothing…The vendors, hardware, protocols, tools, were all completely new to me. I was lucky to receive an opportunity to join an awesome team of networking professionals at an Edu in Brisbane Australia.

My job was to assist with the deployment of 100s of Cisco 2950 switches, upgrading a 10Mbps network to 100Mbps to the desktop. All buildings in the campus were to be re-cabled with Cat5e structured cabling. At cutover we would remove all old equipment and implement the new super fast Cisco kit.

At that point in my career it wasn’t my job to question why we were deploying Cisco. Someone else, probably my boss or even his, had made that call. I got on with mine, working with the senior members of the team to build configs and deploy the equipment. The project was a great success, we upgraded building after building with no issues. The Cisco equipment just worked. I was sold! Cisco was awesome. We deployed 100s of switches with almost no failure.

When it came to basic switching and routing at the Edu it was 100% Cisco. I completely understood the logic. Cisco was solid, we had knowledge of the product and being an Edu we received a kick arse discount so the price didn’t even resemble list. Why wouldn’t you keep installing the number one enterprise networking vendor under those circumstances! It wasn’t until I left the Edu and started working at a VAR that I realised there are other vendors.

My role at the VAR was designing and building networks to support Mitel Voice Systems. My job was to integrate the voice solution with the customers existing network, this meant dealing with a range of vendors including HP, Extreme, Netgear and D-Link. All of these vendors did the job for their customers, voice was clear and my company got paid. If we were required to supply both the network and voice solution we never recommended Cisco. For a small VAR there was no way we could buy and on sell at a competitive price, not to mention maintain the certification level required to be a fully fledged Cisco partner. We found value in other vendor products.

In my current role I’m responsible for recommending products and it’s up to me to ensure the product works both technically and from a value perspective. Alternative vendors have great solutions and it’s about finding what works for your use case. In recent times I’ve recommended moving away from pure Cisco switching and routing solutions to competitors such as Arista and Brocade.

I recently upgraded two data centre networks where after evaluation it was determined that the alternatives best met our challenges. Months later the kit is up and running, frames are being forwarded and packets are being routed. For this use case the competing Cisco solutions would also have done the job, but it was deemed the extra functionality, complexity and cost was not necessary to solve the core problem.

As we move into 2015 I think more focus will be placed on building use case driven, best of breed solutions in networking. Cisco will no doubt continue to exist, but from speaking with colleagues and friends, people are hungry to explore their available options.

Note: I’m not anti Cisco! Over the last years I’ve recommended and installed a range of Cisco solutions, many of which have been successful deployed and continue to work today. I’m for the best of breed solutions, selecting what works for your use case, this includes selecting from Cisco products.