Igniting Innovation: 3 Tips for Intelligent Use of Cloud Services

Cloud is an essential ingredient to enable innovation in your organisation and establish a platform for digital transformation success. The New Normal We’re fortunate to be living in a time of unprecedented change. With a credit card, a laptop, and an investment of time, new start-ups are challenging established businesses and disrupting entire industries. Cloud native organisations are able to achieve technical advantage by leveraging amazing technologies such as artificial intelligence, machine learning, IoT, and intelligent edge services. Public Cloud providers, such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform, enable global scalability and elasticity without stepping into a single data centre or purchasing a single server. Serverless technologies, such as AWS Lambda or Azure Cloud Functions, enable developers to build complex applications without patching or maintaining an operating system. These technologies work, they’re mature, and proven to deliver hyper-scale, real world solutions. Whilst we can now consume advanced technology capabilities as a service, this is not the secret sauce that makes an organisation innovative or successful. I believe the success of innovative, cloud native organisations is drawn from their ability to leverage rapid innovation from public Cloud providers and free themselves from significant technology investments. They leverage Cloud as a platform for innovation to prioritise their customers and focus on providing products and services their customers don’t even know they need yet. It is no longer a question of whether to use public Cloud services, but rather how to leverage Cloud to create an organisational culture focused on delivering the best customer experience and new solutions through smart use of these services. Measuring Success It’s a reasonable question to ask — where is the evidence that Cloud native organisations are more innovative and more successful? A historical look at the top 10 S&P weighted stock is a good place to start. In 1978 the only technology company to make the top 10, ranked by market capitalisation, was IBM. In 2018, technology companies are some of the most powerful organisations in the world, with the likes of Apple, Microsoft, Amazon, Facebook, and AT&T dominating the top 10. In a short 40 year period, technology companies have transformed the American economy through innovation and disruption. Whilst the transformation of the Australian economy has not been as dramatic as that of the United States, new digital native organisations are still making a considerable impact. Atlassian is at the forefront of this, with a market capitalisation of over USD$10B and over 2000 employees since being founded in 2002. One of my favourite Aussie startups, Canva, has also made a significant impact since its foundation in 2012. In six short years, Canva has rapidly become one of Australia’s few ‘Unicorns’; that is, a start-up with market capitalisation valued at more than $1B. Financial metrics alone do not tell the full story of how cloud native organisations are innovating and disrupting. We believe that Cloud and DevOps go hand in hand. Each year, for the past 6 years, Puppet Labs has published a research report with clear evidence that higher IT performance leads to improved business outcomes. Through a productivity lens, high performing organisations spend ‘21 percent less time on unplanned work and rework, and 44 percent more time on new work’. This measure alone contributes to better employee engagement by keeping work interesting, and better profitability by focusing on new activities rather than issue resolution. Cloud as a platform for innovation is still a way off for many organisations. The recently published “Rightscale: State of Cloud” report indicates that among enterprises, respondents are still running ~70% of workloads in non-public Cloud environments. Our Learnings As a business, we too have been transformed by Cloud. From being at the forefront of Aussie start-ups during the .COM boom of the early 2000s, to becoming a provider of smart, creative, digital solutions organisation (and most importantly from working with organisations like yours), we’ve developed 3 quick tips to assist adoption of cloud as a platform for innovation: 1. Unlearn One of the most difficult aspects of organisational change is removing the constraint of “we have always done it this way”. A common theme with high-performing IT organisations is their ability to adopt new technologies and ways of working without constraint. Cloud provides a unique opportunity for enterprises to rethink automation, applied security, service management, and, most importantly, organisation structures and processes. 2. Think Big. Start Small The “State of DevOps” report articulates the importance of leadership vision and establishing a common purpose for your team. We believe that it is equally important to execute transformation following the lean startup method of breaking change down into ‘small batches’ and taking a build, measure, learn approach. In our experience, the twin mantras of ‘constant learning’ and ‘speed over perfection’ lead to the quickest and best long term results. While creating a ‘centre of excellence’ or ‘innovation group’ may assist in commencing your transformation, your long term goal should be to focus on building an innovation culture and avoiding silos. 3. Seek Expertise “There is no compression algorithm for experience”, Andy Jassey, CEO Amazon Web Services. Adopting public Cloud intelligently, breaking down barriers to develop DevOps practices, and creating a culture of innovation are significant changes. The good news is, as an industry, we’re now 10 years into the Cloud journey, and there is a wealth of expertise and learnings you can draw on to assist you in accelerating your journey. Join a local meetup, attend an industry event such as AWS Summit, Microsoft Ignite or Google Cloud Summit, or reach out to partner such as Melbourne IT to discuss your Cloud and innovation aspirations. Start Building Treating Cloud as “someone else’s data centre” will not create an agile organisation that is able to benefit from the changing digital, economic and social environment of tomorrow. The evidence shows that Cloud native organisations are more agile and innovative which directly leads to happier customers, more engaged employees and overall improved business performance. If Cloud is not part of your organisation’s innovation or transformation agendas, your competitors have a clear competitive advantage. Intelligent use of Cloud allows you to focus on your customers and your business, and not building data centres and managing servers. I hope our tips help, in some small way, to spark your interest in trying something new, building something new, or simply thinking about how your organisation can leverage Cloud to innovate.   Shane Day is the General Manager of Cloud, Data, and Analytics for Arq Group’s Queensland team. Shane is passionate about assisting Australia’s leading organisations to deliver digital solutions that enable a great customer experience through smart use of Cloud services. Shane was pleased to be recognised as an AWS APN Cloud Warrior in 2018.   Additional Reading State of DevOps: https://puppet.com/resources/whitepaper/state-of-devops-report Rightscale State of Cloud: https://www.rightscale.com/lp/state-of-the-cloud Ahead in the Cloud: Best Practices for Navigating the Future of Enterprise IT: https://www.amazon.com/Ahead-Cloud-Practices-Navigating-Enterprise-ebook/dp/B07BYQTGJ7    

Security

Augmented Reality: The Inexorable March

  With the advances in technology over the last ten years, it is amazing to see that the retail experience has remained largely unchanged. Although the internet has stolen the echo of a million footsteps, those that still brave supermarkets and clothes shops are mirroring a pilgrimage which hasn’t changed since the dawn of time. They might be self-serving at the checkout but, in essence, this is a domain untouched by technological advancement. As someone who feels a moment of dread, a deer in the shop assistant’s headlights, when they lock eyes and utter those fateful words: “can I help you with anything?”, I am relieved that we might be on the cusp of a new generation of shopping experience. With the advance of augmented reality (AR) on both iOS and Android we are at a point where AR has the ability to extensively change not just the products we buy, but the essence of shopping itself. In this article, I am looking at the impact AR could have on the retail space, ways in which it might affect the consumer, how it works and where we can apply it. I will also be analysing common misconceptions about what AR is and exploring ways of maximising its value. Getting a consumer to take out their phone and use an app is an uphill battle — meaning we, as developers, need to be very discerning in how we approach a solution. What is AR? AR is a technology which enables us to impose a computer-generated image onto a user’s view of the world. Using the real world as a background, objects can be inserted and interacted with. AR can detect horizontal planes, the floor or surfaces, adapting the experience based on where the camera looks, how the user moves, and their physical location in 3D space. If AR is completely new to you, check out this great video which goes over the basic concepts in more detail. How does it work? To fully realise the potential of AR we must recognise both its strengths and weaknesses. For this, we need a rudimentary knowledge on what is going on behind the scenes. Everyone has used the maps functionality on their phone. The application uses your location, which way you are facing, and your elevation to get you to your destination. This works excellently on a large scale, utilising data from multiple GPS satellites, where a ten metre discrepancy only puts you on the other side of the road. For building navigation, a higher level of control is required to know exactly what the user is doing. This is made possible by the recent availability of both hardware and software which combine to provide the significantly higher degrees of freedom and accuracy required by AR. Device rotation allows us to sense the pitch, roll and yaw of the device. In layman’s terms this is the rotation of the device in x, y and z planes of movement.   Pitch, Roll and Yaw on your device   Movement in 3D space is detected using the sway, heave, and surge of the device. These let us know when the device actually moves in those three planes of motion.   Sway, Heave and Surge on your device   Image analysis is the final piece of the puzzle. With the capacity to identify planes and specific shapes, QR codes for example, it gains more information on what it is looking at. AR is the combination of all the above — device tracking, scene analysis and 3D scene rendering. The device knows exactly where you are, which you’re facing and precisely when and how you move. Using this information, it can insert and manipulate computer-generated images to make them appear as though they are fixed in space. The cherry on top is being able to partially understand what it sees — allowing it to add information based on what it is looking at.     Augmented Reality — a door to another world   Armed with this knowledge we are now ready to look at the two biggest innovations that I can see coming to the retail space: Path-finding: Helping users navigate a store to find specific products Enhanced experience: Analysing information the user receives when viewing products Path-finding Path-finding overlays information and route details onto the user’s screen. Using knowledge of the user’s position, objective and obstacles, an optimal route can be shown on their screen. Most of us know the frustration of not knowing where a specific product is located. This might occur when visiting somewhere new or after a shop has rearranged. Similarly, in the shops we don’t visit regularly, we waste time looking for someone to ask for directions — before we even start navigating there. This is where path-finding comes in. The most basic use is being able to guide a user to the product they are after. Having virtual arrows guide them around the store directly to the tinned sardines allows shoppers to quickly navigate to their desired product without needing to call for assistance.   Route-finding through an airport with the American Airlines application Improving efficiency The obvious next step might appear to be utilising this further. To link with a user’s online basket, plotting their most efficient route. At this point we must step carefully with AR, looking at how it can be used to best effect instead of attempting to throw the kitchen sink at it. We get more value by focusing on what the user can’t currently do, using AR to improve this experience, rather than offering too much by trying to fix everything with AR. For this reason, I feel that the technology is best suited to the following simple functionalities: Item location Way-finding truly shines in larger stores or ones not visited often, with consumers wasting significant time finding their bearings and knowing which direction to go. It couples well when searching for a single specialised item, when a direct route in and out of the store is the best outcome for the shopper. Calling for help Many stores save money by hiring less staff, but they reap what they sow when customers need to ask a question and can’t find anyone to help. Path finding enables them to signal for help, see the assistant’s location and see them making their way to you. This frees you up to continue shopping while waiting for the assistant to arrive and alleviates the frustration this scenario usually induces. Setting your point of reference in AR allows you to know your position relative to items or shop assistants   Setting your point of reference in AR allows you to know your position relative to items or shop assistants   Enhanced customer experience: So far, we’ve looked at how we can make the user’s shopping experience better by improving their efficiency finding a specific product. Now we’ll take a look at the possibilities once they find it. With hundreds of products available, consumers have their own unique brand and cost preference. AR allows all of this information to be taken into account. Before we get our teeth stuck in, it is worth understanding a little more about image recognition. This touches on a misnomer around AR. AR uses image recognition to analyse what the camera sees and insert images into this frame. People often assume all image recognition is AR, this isn’t the case. Machine learning and its common use case of image classification can be combined with AR but is considered a completely unique field in its own right. There are two main ways we can utilise image recognition to identify objects: Specific image identification: The technology behind AR can identify certain images. These might be added manually to the project, with information assigned to it, or be a more general class of image, for example a QR code. Whereas QR codes have a certain amount of information baked into their design, manually images can retrieve information from a database once they have been identified.   Specific image identification allows us to display unique AR objects   Image recognition: Companies are doing a huge amount of work on machine learning at the moment. Instead of recognising a specific image, added to an application, it analyses the camera input and identifies what is actually there. It looks at the colours and shapes, using deep machine learning, to identify what it is looking at.   An example of Google Lens in action   It is worth remembering once again our value adding mission statement. What do consumers want that will enhance their shopping experience? Search and filter We have all, at one point in our lives, wished life had a search functionality. CTL+F — “Missing brown sock” An ability that only mums seem to possess when looking for the ketchup in the refrigerator. In our high speed, global environment this is even more apparent. With multiple brands all sitting on the same shelf it is easy to become paralysed by choice. This is where image recognition comes in. This technology can simplify what the user sees, either searching for something particular or identifying additional items that might be of interest. In doing this we remove the superfluous information from the real world, allowing the user to focus on what they really want. Are you a vegetarian? Want to buy more Australian products? Trying to reduce your sugar? These filters can be applied by simply pointing the phone camera at the display shelves, automatically identifying the best products for you. This simple example highlights the strength of this technology. Our brains can only process a finite amount of information at once, while our phone processor can analyse, filter, and display a huge amount of information very quickly. Information enhancement With this window to the augmented world we can provide access to not just more information but targeted information. A tablet or mobile phone allows easy representation of data and comparison of products, something not easy in the real world. This is where image analysis comes in. The ability to identify brands and understand what is being looked at allows us to make predictions on what the user wants to know. For example, looking at a shelf of house plants we could see which require a similar amount of attention and could therefore grow together in the same window box.     We have looked at the strengths of AR. Before we finish we should look at its limitations, too. AR is not a magic bullet. The technology is improving in leaps and bounds, but as developers we still need to take its weaknesses into account when designing a new experience. In our above example a key problem is applying our solution to the real world. Not all stores are the same size or shape, and stock might change location regularly - how will this affect our way-finding? We need our experience to be easily translated to different environments instead of requiring significant work for every new location. The answer depends on the model we base our application on. In this case we would split it into two distinct parts. The first would contain the immutable information. The shape and size of the store, the shelving units and other objects customers can't walk through and won't change position regularly. This would be coupled with a more transitive layer where we would place our stock, advertising boards and other changing objects. This allows us to split up the problems we are looking to solve. We group specific objects into categories and then place these on our immutable map. This allows us to create routes from the user to a specific area. We can then use image recognition to identify the objects they are after and provide detailed information about this object. In this way we can manage AR's limitations by carefully planning the model used before starting development. AR must be used like any other tool - at the right time, in the right way, and with a strong framework of programming supporting it. The enigma behind AR seems to be that, for once, technology is not trailing implementation. We have all the tools but are still trying to understand what it is that consumers want, what they will use, and how the technology will add value to businesses. With developers all over the world learning the language and frameworks behind this technology, we are now waiting for this innovation to change the world and the landscape in which we live. Augmented reality is a door to another world of possibility. The technology is well and truly here and, for the last few years, companies have been hesitating to adopt it. We see flashy videos showing its possibilities but don’t see it in our everyday life. All it will take is the first killer experience to be developed and the floodgates will truly be opened. It will be a breakthrough that will have an elegance and simplicity that will leave the world wondering how we didn't see the wood through the trees. And we don’t just do AR, we can partner you all the way – from strategy, to project management, to analysis and delivery. Just like we’ve done for companies like ANZ, NAB, Coles and Transurban. If we can help you, we hope you’ll let us know.   Simon Smiley-Andrews is an Associate Software Developer with Outware - Part of Arq Group and is responsible for developing applications across a range of projects. He has 5+ years' experience in software development, predominately working on integrated application solutions for a number of organisations within Financial Services. 

Security

Cloud Data Platforms: Not Just a Lift and Shift

  It’s hard to believe that AWS was launched in its current form over 12 years ago, but in this time the service offerings available have evolved massively. For a long time, migrating Data Warehouse and Analytics applications to the cloud made no sense. Most line-of-business (LOB) applications were on-premise and costs (in time and money) of large data transfers into the cloud were prohibitive. That equation is changing, with cloud becoming the ‘new normal’ for enterprise applications. So much so that the 2016 IDG Enterprise Computing Survey found that data and analytics were the leading workloads moving to cloud in 2017. Cloud offers many benefits, including cost, scalability and resilience improvements. Attractively, cloud offers the ability to convert large CapEx outlays to more evenly distributed OpEx over the lifetime of your system, as well as a huge number of new technologies available as SaaS (Software as a Service) that are simply impossible to set up on premise for many businesses, including artificial intelligence and machine learning applications. This is why you should see migration to the cloud as an opportunity to re-architect your workload instead of implementing the same technologies in the cloud. In fact, implementing the same on-premise optimised architecture may cost you more in the cloud! To illustrate this, let’s use an example. We’ll take a very simple data warehouse environment that we will migrate using a couple of different approaches. Let’s suppose that this warehouse: Uses change data capture technology to replicate data Uses a database cluster to store the data Has an ETL process that runs each night to complete batch processes Exposes reports and ad-hoc query capability via a BI Server. This is a very basic configuration that would support a department, however many of the concepts are the same for larger implementations. The key aspects of this architecture are that: All components are on-site and have a fixed capacity – frequently running at that same capacity 24 hours a day Costs incurred are large, up-front, with licencing, support and maintenance ongoing Investment required is relative to the peak capacity provisioned – maybe only fully utilised 2 days a month for month-end processing! In this blog, we will look at a couple of scenarios for migration to a Cloud Environment: Cloud Lift and Shift – Reimplementing the same architecture on cloud infrastructure; and Cloud Re-architecture – Redesigning the capability to make best use of cloud offerings. Cloud Lift-and-Shift Cloud lift and shift is a frequent first step into the cloud as it offers a faster, lower-risk entry point into the cloud. This approach is frequently used for customised LOB applications that have low tolerance for risk and cannot be easily refactored or upgraded. However, is it appropriate for a data and analytics platform? To deliver the same capability with a Lift and Shift approach, we would just re-provision the same servers in the cloud: On-premise CDC servers would be replaced with cloud CDC servers (maybe AWS EC2 instances) On premise database servers will be replaced with cloud database servers And so-on for each layer… This still provides benefits by increasing the speed you can implement Infrastructure changes (independent of the software running on it), rather than the weeks required on premise. However, the architecture is: Likely to have a similar TCO, shifted from up-front CapEx to an ongoing infrastructure OpEx Still running 24/7 and constrained by the size of the servers that you provisioned Still costing you a fixed amount of money (even if no-one is using it or no data is being loaded!) Will still have similar software licencing costs, where licences are required to match peak capacity Still requires management and administration – exactly like an on-premises data warehouse. However, all is not lost. Depending on your appetite for change you can: Save money by turning off certain parts of your architecture overnight (such as your reporting server) Clone the environment for non-production usage much more quickly Turn off development and test environments as required. Cloud Re-architecture Whilst an environment re-architecture will have a higher risk and effort profile, the changes made can deliver significant new benefits to the business. These benefits will include: Offering the ability to scale capacity up and down on demand by moving away from legacy technologies that were not built for rapid scalability – allowing the platform to accommodate expanded user needs faster Delivering a true consumption-based costing model, where costs track consumption closely. Therefore your business will not incur costs 24/7 to provide for peak capacity Reducing the administrative burden by using SaaS or PaaS (Platform as a Service) offerings where every-day administrative tasks like patching, backups and storage management are managed by the cloud provider – allowing you to focus your team budget on higher value innovation or application-level activities. We achieved these benefits by doing the following: Change from a constantly running server for CDC to a Kinesis Firehose service to capture Change Data from our source system – saving money by paying only for the usage required. For example, a stream capable of 1000 x 5 KB records per second will cost US$118 per month (12 TB/month) Changed from a self-managed database in EC2 to a Redshift cluster – optimised for analytics. This reduces management and administration requirements; increases performance for data warehousing workloads; increases availability, recoverability and durability including automated backups; and is scalable without downtime Updated our ETL to run on ephemeral EMR Clusters instead of traditional ETL software on EC2 servers (using Spark, a data processing toolset). These clusters are only instantiated for the duration of batch runs and can be tuned or re-sized as required should data volumes significantly grow or shrink – without the long lead time of the traditional ETL infrastructure it replaces Replaced our reporting solution with Amazon QuickSight to avoid paying for constantly running servers and separate BI tool licences. QuickSight operates as a service that hosts a reporting environment and is priced from US$9 / Month. This is a potentially simplistic example without considering other factors such as the costs of re-factoring or redeveloping some potentially complex components of your warehouse infrastructure, however it demonstrates some of the benefits realisable in even a simple warehousing environment. In summary, to best utilise the true potential of cloud computing, it is necessary to look beyond replication of the status quo in a different environment and truly consider the opportunities that are available to the organisation. Our tips would be: Evaluate serverless services to ensure that you are truly paying for consumption, not for an expensive virtual server to run 24/7 Where serverless architectures are not available, consider offerings that scale readily in response to demand Automate your environments to enable fast provision of new environments for test and development Evaluate if you really need 24/7 Development & Test environments – you can turn off these environments overnight and on non-business days Put in place governance controls to monitor and adjust Cloud spend. Cloud providers have numerous tools to help, but it is important to monitor usage to ensure cost control Modularise your architecture to allow substitution of any one part and develop a roadmap for continuous improvement. For example, AWS last year announced hundreds of new products and services! Working with us means engaging some of Australia's most experienced and skilled cloud architects. It means partnering with dedicated data scientists. And working with people who understand cloud environments inside-out. And it means even the most innovative and interesting technology needs to achieve real-world outcomes.   James Hartley is Head of Data Engineering with Infoready - Part of Arq Group and is responsible for developing the Data Engineering competency nationally. He has 10+ years’ experience in the Information Management industry, predominately working on complex Data Warehousing and Business Intelligence projects and has undertaken roles in Australia and the UK for organisations within Financial Services, Communications, Resources, Gaming, Education and Government.

Security