You know that a term you coined has made it mainstream when people use it regularly in conversations and rarely understand what you meant.
— Martin Fowler (paraphrased from an in-person conversation)
Rouan summarises DevOps culture well in his post on Martin’s bliki. It is easy for developers to get disinterested with operational concerns. “It works on my machine” used to be a common phrase between developers in yesteryears. Some operations folks can also be less concerned by development challenges. Increased collaboration can help build a bridge in the gap between Developers and Operations team members and thus make your product better.
This increased collaboration has made observed requirements like system and resource utilisation monitoring, (centralised) logging, automated and repeatable deployments, no slow-flake servers etc. key parts of our products. Each of these improve the quality of life of your product either by directly benefiting the end user or making the system more maintainable for Developers and Operations users thus reducing the time to fix issues for end user issues. Developers and Operations folks are also first class users of your system. Their happiness (ease of debugging issues, deploying etc.) is a key part of your product’s success. It allows them to spend more time improving your product for paying end users.
MLOps is a culture that increases collaboration between folks building ML models (developers, data scientists etc.) and people who monitor these models and ensure everything is working as intended (operations). The observed requirements in your system will have some overlaps with what we have already talked about like system and resource monitoring, (centralised) logging, automated and repeatable deployments, automated creation of repeatable (non-snowflake) infrastructure etc. It will also include a few Data Platform specific observed requirements such as model and data versioning, data lineage, monitoring effectiveness of your model over an extended period of time, monitoring data drift etc.
The need of every data platform is slightly different based on the challenges you are solving and the scale at which you operate. One of the platforms I’ve been working on produces 2TB of data every week. It didn’t take too much time for data storage costs to be the number 1 line item on our bill and we invested some time in optimising our storage and retention strategy. Other teammates have lowered data volumes and focus on reducing the cycle time for model creation. Your mileage may vary.
Based on our experience building data platforms over the past few years, here are a few tools we have used and things we have watched out for.
Choose a storage mechanism that provides cheap and reliable access to your data while meeting all legal requirements for your dataset. If you are in a heavily regulated environment (finance, medicine etc.), you might not be able to use the cloud for customer data. The techniques still remain similar. Partition your data based on access requirements and retention times. Archive data when you do not need it. Use features like push down predicates to efficiently read your data.
We recently wrote about data storage, versioning and partitioning which goes into great depth into this topic.
Your data pipelines will get complex over a period of time. Much like infrastructure as code, we would like our data pipelines in code. Apache Airflow is one of the tools that allows us to do this fairly easily. Sayan Biswas wrote about our airflow usage in 2019. Over the last few years, we have made dozens of improvements to the way we use Airflow. In a subsequent post in this series, we will talk through these improvements.
We spawn EMR clusters on demand and terminate them when jobs complete. A cluster runs only 1 spark job (and a few extra tasks for cleanups and reporting). If a job fails due to resource constraints, this helps isolate if another hungry job consumed too many resources before a scaling policy kicked in.
Each EMR cluster has an orchestrator node (AWS and Hadoop call them “master nodes”) and a group of core nodes (Hadoop calls them “worker nodes”). We request for on demand nodes for orchestrators and reserve the instances to reduce cost. We bid for spot instances for cores using a dynamic pricing strategy that is dependent on the current price. We have considered building a system that automatically switches instance types based on availability, price and stability in AWS but failures in spot bids are currently rare enough that it does not justify the cost of developing this feature.
We also monitor the resource utilisation of our spark jobs using Ganglia on AWS EMR. This tells us our CPU, memory, disk and network utilisation for our clusters. Since the information on Ganglia is lost when clusters are terminated, we run an EMR step to export a snapshot of Ganglia before the cluster terminates. This in conjunction with persisted spark history server data on AWS allows us to tune underperforming spark jobs. In a subsequent post, we will go into details of how to monitor your jobs effectively and tune them.
Airflow creates EMR clusters and monitors each of the jobs. If a job fails, Airflow notifies us on a specific slack channel with links to the Airflow logs and AWS cluster.
Complex spark applications produce hundreds of megabytes of logs. These logs are distributed across the cluster and will be lost when the cluster is shut down. AWS EMR has an option to automatically copy the logs to S3 with a 2 minute delay.
We have tried using CloudWatch to index and analyse our spark logs but it was far too expensive. We also tried using a self hosted ELK stack but the cost of scaling it up for the volume of logs sent was too high. Dumping it on S3 and analysing it offline gave us the best cost to performance ratio.
To help reduce the time to fix an issue, when an issue is detected, the EMR cluster analyses its logs from YARN and publishes an extract onto slack as an attachment. Any further detailed analysis can be done on the logs in S3.
Every time we write code, we run tests to ensure the code is safe to be deployed. Why don’t we do the same thing with data every time we access it?
When you first look at the data and build the model, you ensure the quality of the data used for training the model meets acceptable standards for your solution. Data Quality is measured by looking at the qualitative and quantitative pieces of your dataset. Over a period of time, these qualitative and quantitative attributes might drift causing adverse effects on your model. Thus, it is important to monitor your data quality and data drift. Data drift might be large enough that your model does not produce the right results any more or might be small enough to introduce a bias in your results. Monitoring these characteristics is key to producing accurate insights for your business.
Tools like Great Expectations and Deequ will ensure that your data is sound structurally and volumetrically. Deequ also has operators to look at rate of change of data which is a better expectation than having static thresholds on large volumes of data.
For example, given an employee salary database where the salary is nullable, a check to ensure no more than 100 employees out of the 1000 you currently have data for have no reported salary is bound to fail when the data volume increases significantly. If this check was to ensure no more than 10% of employees have no reported salary will work as the data scales as long as it scales evenly. Moving to a check that looks at rate of change of ratio of users not reporting a salary will be more robust. If the number changes significantly (up or down), it might mean that it’s time to tune your model since the source data is drifting away from when it was trained.
There are more complex examples on how we watch for data drift that will have to wait for a dedicated post.
When our end users feel pain, we add new features to make their experience better. The same should be true for developers/operations experience (DevEx/OpsEx).
When it takes us longer to debug a problem or understand why a model did what it did, we improve our tooling and observability into our system. When it ran slower or was more expensive, we improved our observability to investigate inefficiencies quicker.
This has allowed us to grow our data platform 10x in terms of features and data volumes while reducing the time taken to produce insights for our end users by 98.75%, the cost to do so by 35% and not to mention a significant improvement in developer and customer experience.
Thanks to Jayant, Priyank, Anay and Trishna for reviewing drafts and providing early feedback. As always, Niki’s artwork wizardry is key!
]]>In this post, we are going to use the terminology of AWS S3 buckets to store information. The same techniques can be applied on other cloud, non cloud providers and bare metal servers. Most setups will include a high bandwidth low latency network attached storage with proximity to the processing cluster or disks on HDFS if the entire platform uses HDFS. Your mileage may vary based on your team’s setup and use case. We are also going to talk about techniques which have allowed us to efficiently process this information using Apache Spark as our processing engine. Similar techniques are available for other data processing engines.
When you have large volumes of data, we have found it useful to separate data that comes in from the upstream providers (if any) from any insights we process and produce. This allows us to segregate access (different parts have different PII classifications) and apply different retention policies.
We would separate each of these datasets so it’s clear where each came from. When setting up the location to store your data, refer to local laws (like GDPR) for details on data residency requirements.
Providers tend to make their own directories to send us data. This allows them to have access over how long they want to retain data or if they need to modify information. Data is rarely modified but when it is, a heads up is given to re-process information.
If this was an event driven system, we would have different event types suggesting that the data from an earlier date was modified. Since the volume of data is large and the batch nature of data transfer on our platform, verbal/written communication is preferred by our data providers which allows us to re-trigger our data pipelines for the affected days.
Most data platforms either procure data or produce it internally. The usual mechanism is for a provider to write data into its own bucket and give its consumers (our platform) access. We copy the data into a landing bucket. This data is a full replica of what the provider gives us without any processing. Keeping data we received from the provider separate from data we process and insights we derive allows us to
The data in the landing bucket might be in a format sub optimal for processing (like CSV). The data might also be dirty. We take this opportunity to clean up the data and change the format to something more suitable for processing. For our use case, a downstream pipeline usually consumes a part of what the upstream pipeline produces. Since only a subset of the data is read downstream by a single job, using a file format that allows optimized columnar reads helped us boost performance and thus we use formats like ORC and parquet in our system. The output after this cleanup and transformation is written to the core bucket (since this data is clean input that’s optimised for further processing and thus core to the functioning of the platform).
While landing has an exact replica of what the data provider gave us, core’s raw data just transforms it to a more appropriate format (parquet/ORC for our use case) and processing applies some data cleanup strategies, adds meta-data and a few processed columns.
Your data platform probably has multiple models running on top of the core data that produce multiple insights. We write the output for each of these into its own directory.
Partitioning is a technique that allows your processing engine (like Spark) to read data more efficiently thus making the program more efficient. The most optimal way to partition data is based on the way it is read, written and/or processed. Since most data is written once and read multiple times, optimising a dataset for reads makes sense.
We create a core bucket for each region we operate in (based on data residency laws of the area). For example, since the EU data cannot leave the EU, we create a derived-bucket in one of the regions in the EU. Under this bucket, we separate the data based on the country, the model that’s producing the data, a version of the data (based on its schema) and the date partition based on which the data was created.
Reading data from a path like derived-bucket/country=uk/model=alpha/version=1.0
will give you a data set with columns year, month and day. This is useful when you are looking for data across different dates. When filtering the data based on a certain month, frameworks like spark allow the use of push down predicates making reads more efficient.
We change the version of the data every time there is a breaking change. Our versioning strategy is similar to the one talked about in the book for Database Refactoring with a few changes for scale. The book talks about many types of refactoring and the column rename is a common and interesting use case.
Since the data volume is comparatively low in databases (megabytes to gigabytes), migrating everything to the latest schema is (comparatively) inexpensive. It is important to make sure the application is usable at all points and that there is no point at which the application is not usable.
When the data volume is high (think terabytes to petabytes), running migrations like this is a very expensive process in terms of the time and resources taken. Also, the application downtime during the migration is large or there’s 2 copies of the dataset created (which makes storage more expensive).
Let’s say you have a dataset that maps the real names to superhero names that you have written to model=superhero-identities/year=2021/month=05/day=01
.
The next day, if you would like to add their home location, you can write the following data set to the directory day=02
.
Soon after, you realize that storing the real name is too risky. The data you have already published was public knowledge but moving forward, you would like to stop publishing real names. Thus on day=03
, you remove the real_name
column.
When you read derived-bucket/country=uk/model=superhero-identities/
using spark, the framework will read the first schema and use it to read the entire dataset. As a result, you do not see the new home_location
column.
Asking Spark to merge the schema for you shows all columns (with missing values shown as null
)
As your model’s schema evolves, using features like merge schema allows you to read the available data across various partitions and then process it. While we have showcased spark’s abilities to merge schemas for parquet files, such capabilities are also available with other file formats.
Sometimes, you evolve and improve your model. It is useful to do parallel runs and compare the result to verify that it is indeed better before the business switches to use the newer version.
In such cases we bump up the version of the solution. Let’s assume job alpha v1.0.36 writes to the directory derived-bucket/country=uk/model=alpha/version=1.0
. When we have a newer version of the model (that either has a very different schema or has to be run in parallel), we bump the version of the job (and the location it writes to) to 2.0 making the job alpha v2.0.0 and it’s output directory derived-bucket/country=uk/model=alpha/version=2.0
.
If this change was made and deployed on 1st of Feb and this job runs daily, the latest date partition under model=alpha/version=1.0
will be year=2020/month=01/day=31
. From the 1st of Feb, all data will be written to the model=alpha/version=2.0
directory. If the data in version 2.0 is not sufficient for the business on 1st Feb, we either run backfill jobs to get more data under this partition or we run both version 1 and 2 until version 2’s data is ready to be used by the business.
The version on disk represents the version of the schema and can be matched up with the versioning of the artifact when using Semantic Versioning.
Applications, system architecture and your data always evolve. Your decisions in how you store and access your data affect your system’s ability to evolve. Using techniques like versioning and partitioning helps your system continue to evolve with minimal overhead cost. Thus, we recommend integrating these techniques into your product at its inception so the team has a strong foundation to build upon.
Thanks to Sanjoy, Anay Sathish, Jayant and Priyank for their draft reviews and early feedback. Thanks to Niki for using her artwork wizardry skills.
]]>Before we can do that, it’s important to understand build process before we began on this journey.
Our build model for this project was branch based. Each environment maps to a branch (main -> dev
, uat -> uat
and production -> production
). All other (feature) branches only ran the plan stage against the dev
environment.
As you can notice, the configurations, secrets and keys are all maintained on the build agent. This means, every developer wanting to run plan and test their changes needs to replicate the terraform_variables
directory. Any mistakes in doing so masks actual issues that your pipeline might face leading to delayed feedback.
Next, let’s look at what our codebase looked like
The provisioning scripts help us consistently run different stages across modules. Each module is an independent area of our infrastructure (such as core networking, HTTP services etc.)
Each of the provisioning scripts accepted a WORKSPACE_NAME
(branch for execution that maps to the environment terraform is running for) and MODULE_NAME
(module being executed).
init.sh
ran the terraform init
stage of the pipeline downloading the necessary plugins and initializing the backend
plan.sh
ran the terraform plan
stage allowing users to review their changes before applying them.
apply.sh
applied the changes onto an environment. Developers do not run this command from local to ensure consistency on the environment
We moved the variables into the config
directory by making a directory for every branch for each of the 3 environments we had.
According to terraform’s documentation, you can export a variable that your terraform codes need with a prefix of TF_VAR
.
functions.sh
provides convenience functions to read the configuration and secrets.
fetch_variables
read the tfvars
file, removes empty lines (that were added for readability), prefixed the name with TF_VAR
and joined all entries into a single line. The string this method returns can be used as a prefix to the terraform
command while running plan
and apply
making them environment variables.
Updated plan and apply scripts are placed in the secrets management section for brevity
The only limitation is that none of these variables can have a hyphen in the name because of shell variable naming rules. As with any potential mistake, a test providing feedback helps protect you from run time failures. test_variable_names.sh
does this check for us.
Secrets like passwords can be version controlled in a similar way though they require encryption to keep them safe. We’re using OpenSSL with a symmetric key to encrypt our secrets. Each secret is put into a tfsecrets
file (internally a property file just like tfvars
files for configuration). When encrypted, the file will have an extension of .tfsecrets.enc
. When the plan
or apply
stages are executed, files are decrypted in memory (and not on disk, for security reasons) and used the same way.
functions.sh
gets a new addition to support reading all secrets
The astute amongst you probably noticed that we’re using OpenSSL v1.0.2s because v1.1.x changes the syntax on encryption/decryption of files. Also, you might have noticed the use of environment variables like SECRET_KEY_main
, SECRET_KEY_uat
and SECRET_KEY_production
as the encryption keys. These values are stored on our CI server (in our case GitLab) which makes these values available to our CI agent during execution.
For local development, we have scripts to encrypt and decrypt configuration files either one at a time or in bulk per environment. It’s worth noting that re-encryption of the same file will show up on your git diff
since the encrypted file’s metadata changes. Only check in encrypted files when their contents have changed (helping you debug future issues)
encrypt.sh
takes SECRET_KEY
as an environment variable for making local usage easier.
decrypt.sh
also takes the same SECRET_KEY
as an environment variable for making local usage easier.
If all files for an environment aren’t checked with the same key, you’ll face a runtime error. Since files can be encrypted individually, you must test if all files have been encrypted correctly. This test is also useful when you’re rotating the SECRET_KEY
for an environment.
test_encryption.sh
needs SECRET_KEY_<env>
values set so it can be executed locally.
Our final project structure contains the following files
terraform
├── config
│ ├── main
│ │ ├── module-1.tfvars
│ │ ├── module-1.tfsecrets.enc
│ │ ├── module-2.tfvars
│ │ └── module-2.tfsecrets.enc
│ ├── production
│ │ ├── module-1.tfvars
│ │ ├── module-1.tfsecrets.enc
│ │ ├── module-2.tfvars
│ │ └── module-2.tfsecrets.enc
│ ├── uat
│ │ ├── module-1.tfvars
│ │ ├── module-1.tfsecrets.enc
│ │ ├── module-2.tfvars
│ │ └── module-2.tfsecrets.enc
├── module-1
│ └── ...
├── module-2
| └── ...
└── scripts
├── decrypt.sh
├── encrypt.sh
├── provision
│ ├── apply.sh
│ ├── functions.sh
│ ├── init.sh
│ └── plan.sh
├── test_encryption.sh
└── test_variable_names.sh
plan.sh
uses functions.sh
to load configuration and secrets
apply.sh
uses functions.sh
in a similar fashion
And thus, our terraform project requires no data from the CI agent and can be executed perfectly from any box as long as it has the latest code checked out and the correct version of terraform.
]]>commit.gpgsign = true
by using
git config --global commit.gpgsign true
What if you have different signatures for your personal ID and your work ID?
First, you create multiple signatures. It is important that the email address in the signature is the same as the one for the user who has authored the commit. Run gpg -K --keyid-format SHORT
to see all available keys. The output looks like
/Users/karun/.gnupg/pubring.kbx
-------------------------------
sec rsa4096/11111111 2019-06-11 [SC]
1234567890123456789012345678901211111111
uid [ultimate] Karun Japhet <karun@personal.com>
ssb rsa4096/22222222 2019-06-11 [E]
sec rsa4096/33333333 2019-06-11 [SC]
0987654321098765432109876543210933333333
uid [ultimate] Karun Japhet <karunj@work.com>
ssb rsa4096/44444444 2019-06-11 [E]
Fetch the ID for each of the signatures. The ID for the personal signature is 11111111 and that for the work signature is 33333333. To assign a signature to the repo, execute git config user.signingkey <ID>
.
Personally, I have aliases for personal and work signatures and every time I checkout a project, run the alias once.
alias signpersonal= "git config user.signingkey 11111111 && git config user.email \"karun@personal.com\""
alias signwork = "git config user.signingkey 33333333 && git config user.email \"karun@work.com\""
Run git log --show-signature
to verify if a commit used the right signature. Happy commit-signing.
If your login just won’t work, try changing the following settings
Allow calls to accounts.google.com
& apis.google.com
Allow Third party trackers in Firefox through Settings > Privacy & Security > Cookies > Third-party trackers
]]>If the app does not ask you to hit the button on your bridge, your account already has a bridge associated with it.
You can see what bridge is associated with your MeetHue account on the bridges page.
Remove any older bridges you might have on your account and try logging into the Phillips Hue app again. Once complete, you should be able to link your Google Home assistant to your Phillips Hue app.
You can cleanup how many apps have access to your account and how many other users have access to your bridge. If you see anything that your don’t recognize, remove it. After all, these apps and Hue account users can control the lights in your house. If you don’t know them, remove their access.
In my case, the only users on my bridge are the family members in my house and the only apps I have are the Phillips Hue Android app (for mobile access remotely) and Google (assistant integration).
]]>Most applications these days should have a single (console) appender. This can be linked up with your log aggregator of choice. If your application cannot aggregate logs off the console stream, file is your next best alternative.
Wrap each of your appenders with an async appender and add the async appender to your root logger.
Every call to the logger creates a log event. In synchronous logging, that log event was processed and writes were made to all appender streams before the application continued. Since most stream writes involve I/O, this meant the application would wait for I/O before continuining thereby slowing it down. With async logging, the event gets pushed to a log level specific in memory queue. These events are processed and consumed by the appenders asynchronously. Since the application can continue after a log event has been published to the queue, asynchronous logging works quicker (as long as I/O is the long pole in the tent that is publishing log messages)
Here’s a sample configuration:
<configuration>
<appender name="FILE" class="ch.qos.logback.core.FileAppender">
<file>myapp.log</file>
<encoder>
<pattern>%logger{35} - %msg%n</pattern>
</encoder>
</appender>
<appender name="ASYNC-FILE" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="FILE" />
<queueSize>1024</queueSize>
<neverBlock>false</neverBlock>
</appender>
<root>
<appender-ref ref="ASYNC-FILE" />
</root>
</configuration>
Every queue has a configurable depth. The depth of the queue is based on how much memory you have and expected ratio in rates of messages coming in through the application and the messages being published through the I/O bottleneck.
If you hit max queue depth on either the WARN
or ERROR
queues, further statements for those levels become synchronous.
If you hit more than 80% of the max queue depth on any other level, the system will start dropping log statements (due to discardingThreshold=20
by default and neverBlock=true
). Therefore, under high load, you can lose INFO
, DEBUG
and TRACE
log messages. This behaviour is acceptable for most cases except specific critical statements (like audit logs). For such cases, you can add asynchronous appenders that are allowed to block.
The percentage of depth after which messages are dropped is configurable. You can make info/debug logs synchronous at 100% too if needed by changing the neverBlock=false
(which is the default behaviour).
All of this information is available on logback’s documentation.
Async logs only work more efficiently because the production of events is synchronous (and hopefully a quick task) and the processing of events (which requires IO) is a slow task.
However if production of log messages takes long time, async logging will not make things better. When you’re printing a large amount of data or if the creation of the log message is an expensive operation, use the following kind of log statement
// style 1: java string interpolation; inefficient and hard to read :P
logger.info("Large object value was " + largeObject1 + " and long operation printed " + largeObject2.longOperation())
// style 2: scala string interpolation; inefficient but easy to read
logger.info(s"Large object value was $largeObject1 and long operation printed ${largeObject2.longOperation()}")
// style 3: logback based string interpolation; efficient but inconvenient to read
logger.info("Large object value was {} and long operation printed {}", largeObject1, largeObject2.longOperation())
While the scala interpolation (style 2) is the easiest to read, we should only do it when the objects being printed are small (small-ish strings or primitives).
Rule of thumb:
Use lazy logging. It internally uses loggers that wraps yours code (during compile time) with if checks to not process log statements if the specific log level doesn’t need to be printed (using macros). Worried about performance due to extra if conditions? You shouldn’t. Modern processors contain black magic called branch prediction that reduce the effect of statements such as this to be effectively nothing.
IMO, every scala project should use lazy logging. It’s light on dependencies and has a nice implementation that makes your logging more efficient run faster for fractionally slower compilation.
]]>On one of our recent teams, our showcases had challenges. Each of these challenges is a piece of feedback. We added structure to our showcases by running it like a theatre recording TV shows.
This isn’t revolutionary stuff. This is an attempt at defining a structure that should make it easier to organize showcases based off a check-list.
The MC is the face of the operations. They are responsible to dessiminate information and keep the crowd engaged. This means that the person should have context about what goes on and how to handle the different failures around client infra (skype issues, VDI issues etc).
Running commentary: Always keep speaking. Is there an issue? Keep the show rolling. Be transparent. Your support (folks below) will keep feeding you information when necessary.
This is the magician that controls the lighting on stage. This person actually runs the slides and the demos ensuring everything is smooth
This is the person who runs the show. This person is responsible to stay on the demo co-ordination chat and spot issues and handle them before they become a thing. This person is also responsible to give instant feedback to people running the showcase when needed.
The person who watches the logs and statuses for the services involved in the demo. If there is anything going wrong, talk to the conductor immediately.
This person is in the room (with clients) and is responsible to keep time. If the discussion goes off, it is your responsibility to cut the discussion off and setup a followup discussion.
If the clients are in multiple locations, have a timekeeper per location. Might be the conductor when available in a location.
Multiple people taking notes and sharing them after the demo. They are responsible to pick up body queues from the people around them and take notes on follow up discussions that we need to have.
Be active on a demo co-ordination chat channel and provide instantaneous feedback from different locations. This helps the conductor get more information and is key to their effectiveness.
This person is primarily responsible for the content of the showcase.
The content of a showcase should be like a TV show. A major milestone/deliverable is like a season and should have an overarching story (aka narrative arc). Each showcase is like an episode and should have a subsection of the narrative arc.
The way Presentation Patterns book describes narrative arcs in presentations is true about showcases
PresentationsShowcases are a form of storytelling; don’t ignore a few thousand years of oratory history. A Narrative Arc is a common trope; organizing yourpresentationshowcase in a similar way leverages your audience’s lifetime of story listening experience.
Know the people on your team. Identify which team members can do what roles. Invest in and groom people for roles based on their interest, it’s a growth opportunity.
If you’re using these tools and would like to upgrade all of the applications you have, run the following command.
brew update && brew upgrade && (brew cask outdated | cut -f 1 -d " " | xargs brew cask reinstall) && brew cleanup
brew update
brew upgrade
brew cask outdated | cut -f 1 -d " " | xargs brew cask reinstall
brew cask outdated
cut -f 1 -d " "
xargs brew cask reinstall
brew cleanup
Note: brew cask cleanup
is now deprecated.
Java is a verbose language. No one disputes it.
Despite the clunky nature of the language syntax, it still is the language of choice in most enterprises. If you work in the services industry or are a technology consultant, chances are that you have to work with Java on a regular basis.
If you’re also a fan of functional programming language and have worked any modern programming language, you’ll recognize that Java’s syntax hinders your productivity because of the large amounts of boilerplate the language will generate. While newer JVM based lanaguages like Kotlin solve these problems in different ways, the open source community created Project Lombok to provide similar syntactic sugar in the world’s most popular enterprise programming language.
Lombok is a Java dependency that uses Java annotations to generate byte code straight into the class files during the compilation phase there by allowing the boilerplate code from your codebase to be significantly reduced.
An example from the Software Engineering Trends post from Jan 2010 shows
would generate the same code as
You shouldn’t have to write code that can be generated automatically. Of course, modern IDEs will do this for you with a few clicks of the keyboard
We’re trying to optimize more than a few clicks though. Have a look at the equals method below:
Is this a standard equals method (one where every field in the class is checked for equality)? Did we skip a field? Did we do a non standard check on one of the fields? Unless you go through the method line by line, there is no way to know.
Generating code saves you the hassle of checking. If there is an annotation, you know the what the implementation will be (assuming you know how the framework works). If there’s code, chances are that it’s a non-standard implementation (or someone made a mistake).
If you wish to check the generated code, you need an IDE that decompiles byte code or a tool that does the same.
If something’s wonky, debugging the issue might not be straight forward.
Modern IDEs like IntelliJ are built for refactoring. One of the most common refactoring options is the option to Change Signature. It’s an extremely useful option that allows you to reorder method (or constructor) parameters and the IDE takes care of the appropriate changes throughout the codebase.
The order of the constructor parameters in a lombok-fied class is the order in which the parameters are declared. Changing this order changes the constructor signature.
For a class with different parameter types, this is not a problem. Refactoring the following class
to the following signature
is not a problem. The usage of the constructor will fail to compile and provide feedback.
If you have primitive types in your lombok-fied class, you have a problem. Refactoring the following class
to the following signature
will provide no feedback. The code will compile and set employeeId
s to firstName
s, firstName
s to lastName
s and lastName
s to employeeId
s. If you don’t have tests on the behavior of the Person
class, you won’t notice this issue until it’s too late. Hopefully, you don’t have tests for a data container with no behavior.
If your team can answer yes to all of the above, you should use Lombok.
I must admit, most large teams can’t answer yes to all of the questions. Have you considered using Kotlin instead? :)
]]>Why? S3 is free for the first year. Even post that period, my bills have been <$0.02/month which is a 99.951% reduction in cost.
Snap CI is will integrate with your publically accessible GitHub repositories for free and trigger builds on commit! Connect to your github repository and get it to compile your markdown into html. Deploying to S3 is a piece of cake. Congratulations on having continuous delivery for your blog!
Cloudflare provides the free SSL and Amazon S3 provides the near free hosting. A few cents a month to host your entire website is a good deal!
I’ve been on S3 for a year now and I couldn’t be happier!
** Goodbye Servers, Welcome S3 **
]]>Unit testing is all about focusing on one element of the software at a time. This unit is called the often called the ‘System Under Test’ (refer Mocks Aren’t Stubbs). In order to test only one unit at a time, all other units need to not be test at the same time. As obvious as that sounds, it’s easy to miss.
Classes do not exist independent of one another. They usually have dependencies. Such dependencies are called the ‘Collaborators’. There are multiple ways to manage collaborators that have been talked about by Martin in his article.
Before we go on, please ensure you’ve read through Mocks Aren’t Stubbs by Martin Fowler. This post assumes that you’ve gone through the article before continuing on to commonly made mistakes in Unit Testing
Consider a board game where the Board class runs the game with the help of it’s collaborators Player
and Dice
.
If we consider the Board
to be the System Under Test, the most tempting trap to fall into is start testing the Board directly.
This is not the greatest example but it does attempt to show you the coupling between the different components. Player1’s current position isn’t predictable since it’s coupling with dice. The dependency also means that if the dice has defects, the board can’t be tested appropriately.
By swapping out player and dice instances with mocks, we have the ability to only test the board independent of potential issues with the dependencies.
The above test can be refactored to look like
The test now allows you to check if player1
was moved 3 places since the response provided by the dice is in your control. Mocks also allow you to test that player2
was not called.
This becomes even more important in an example where the response from the mock affects the system under test. Controlling the mock allows you to control predict the end state of the system under test with the assumption that your mock setup is correct. These assumptions can be validated with the spec for the individual mocks. The unit test for dice mock can confirm that the dice only returns values between 1 and 6 (inclusive).
Every functionality should be tested within it’s boundaries. Let’s take the Dice
class as an example and talk about what this means.
Typically a dice produces values between 1 and 6.
It’s corresponding test has to prove that rolling a dice always results in a value between 1-6.
This test proves that the value is inside the range but does not prove that it will always be in that range. Since the implementation contains a PRNG, the end result cannot be predicted.
Most readers wouldn’t have noticed the defect in the implementation.
The implementation can produce values 0-6. The fact that your test passed proves that it is a flaky unit test. The test has a 1/7 chance of failing. The fact that it didn’t fail when you ran it is not surprising :)
The anti-pattern to take away from the previous example is that the Dice class relies on a library and that the library is contained in the class. The fact that it can’t be injected means that you can’t control it.
Dependency Injection is your friend!
Now, your test can work with a mocked Random
instance for more accurate results.
We’re currently making 2 assumptions on the collaborator.
random.nextInt
is always called with parameter 5
random.nextInt(5)
always returns values between 0 and 5The first assumption is in part validated by the mocking library. If Dice
called by any other parameter, the results wouldn’t be what we want. But if you want to be extra sure, you could always make the test fail using an argument captor
The second assumption should not be validated by you. If you look at the documentation for random.nextInt()
you will notice
It is the responsibility of the library (java.util.Random
in this case) to test itself.
How do I know Random
will not misbehave? I don’t. The Dice
component could be integration tested. It is an absolute necessity if you deem the component to be an untrusted collaborator. If this was a database connection or a REST call, you’d want that. For a Java util or a well tested open source library, you could be forgiven for not writing an integration test.
In this case, I won’t be writing one for sure! ☺
]]>I also realized that Kimsufi came up with cheaper servers (now as low as €4.99). Without automation to setup my servers, the thought of migration and building another snowflake server scares me purely because of it’s frailty. No more.
I’ve now officially moved to my new server, Cybershark!
I have made it a point not to have PHP on this server. It’s time to stop my dependency on it. Good bye Wordpress. Welcome Octopress. Good bye Soccer Scraper. Google Now will do fine :)
This means some of the internal dependencies I had (such as PHP mailers) needs to now be moved to another technology. Hello Node.JS. I think we might be good friends :)
]]>There’s a well written reddit page by user hazehk which lists all the setting changes required. Takes around 10-15 minutes to run through them.
If you’re a bit lazier, you could use tools to do it for you if you aren’t paranoid about the tools themselves :)
Fix my Mac OS X talks about the simple changes required to alleviate your pain. You could either follow their steps or use their python script. Your choice!
It has only 2 steps so it should take you less than a minute to do both. I actually went as far as to disable Spotlight all together (remove spotlight shortcut; disable spotlight indexing causes problems with Alfred because it needs spotlight’s cache for application data) and move to Alfred (not the least of which was motivated by the privacy issues..)
]]>We figured that compilations overwriting class files were OK but having to edit any resource just took too long. In such cases, you can use maven’s process-resources plugin to ask maven to only copy the new resources to your target directory.
This is significantly faster. Package times went down from 6 minutes to 18 seconds. Of course, a SSD would have helped but looking at the difference, it’s well worth the effort :)
Go try it out!
]]>A code base which used to take 22 minutes to compile went down to 3 minutes. This just goes to show the effect that disk IO bottlenecks can have on your system.
RAM is significantly faster than spinning platter HDDs. Until the commercialisation of static state devices, it wasn’t even a competition. It’s meant to be like that.
If you need higher writes speeds to disk which don’t need to be persisted over a long period of time, why not use some of your spare RAM as a hard disk? This can be achieved using software.
RAM drive softwares are available for all major operating systems.
SoftPerfect’s RAM Disk is a good free tool for non-commercial processes for Windows. Linux has tempfs and ramfs.
Most big projects follow a multi-module POM structure. You can move your target directory in 2 ways.
You can change the output directory to the desired path ensuring all compiled files go to your RAM drive. Just make sure you qualify your path well using the group and artifact IDs to ensure different projects don’t overwrite each other’s compiled code.
The issue with this approach is that all users on the team are now bound by
The first issue can be fixed by creating a property for the base path. This parameter can be passed as a compile time parameter with a default set in the POM
The second issue can’t be fixed using this approach. This can be fixed with the second approach
Maven allows you to have profiles for applying configs in specific scenarios.
Now you can compile your code with a profile and it will use the specified directory for compilation
Congratulations, your code is compiled on RAM drive. Is it still not fast enough? Is the installation process slow? Well, you could move your M2 directory to the ram drive too
You can change your maven configuration to ask it to move your local maven repository home (which is the place where all the artifacts you build and/or download are stored).
Update your settings.xml with the location of your local repository
]]>Side note, I recommend every user have HTTPS everywhere installed on every browser.
Though it’s not perfect, you can get a SSL for your website for free.
The communication between you and a website looks something like this without SSL.
It’s prone to a Man in the middle attack. This could be done by your neighbour tapping into the line, your ISP or someone half way across the world listening in on the server you request data from.
CloudFlare is a CDN that provides you with additional security by analysing requests using crowd sourced data over hundreds and thousands of websites. Add your domain to CloudFlare and ask your DNS provider to send all requests to the CloudFlare servers. Once the setup is complete, your data will be sent to your server via CloudFlare. It ensures your IP is not exposed outside there by providing it with some amount of Denial of Service attack prevention since your IP is not directly exposed and it is CloudFlare’s job to handle in coming Denial of Service attacks (once set up).
Users must note that the move to CloudFlare means you can’t SSH to your machine anymore because they do not forward the port. You can create a subdomain that doesn’t route traffic through CloudFlare. That makes it easier to SSH/FTP into the box but provides a way for attackers to access your machine bypassing CloudFlare’s security. Alternately you can add a vhost entry on your machine ensuring you can connect with ease but this won’t help you if you would like to connect from some other machine. You could just remember the IP if you’re a pro :P The choice is yours!
CloudFlare provides a universal SSL for all domains routed through their service. As long as you trust CloudFlare’s SSL keys not to be leaked (if they do, bigger businesses would have a problem way before your website does).
This feature is available for free to all users. I recommend using at least the Flexible SSL which requires no further setup. Turn it on and you can hit your website using https. For now, choose Flexible SSL.
Ideally you want HTTPS to be an option your users choose. You’d want to at least make it a default. If people don’t know, they wouldn’t move to using it. I see no reason why browsers these days won’t be showing your content correctly if they are in HTTPS so I recommend having it on as a default.
If you run a blog or website where there are no “users” but readers, it’s hard to let them choose their HTTPS settings. If you haven’t noticed already, you are accessing this website using HTTPS. Welcome to the dark side.
No need to write a htaccess or similar config on your server. Go to CloudFlare and create a HTTP rule for your base domain asking for a redirect to HTTPS.
Not quite.
The communication between you and CloudFlare is secure. The communication between CloudFlare and your server isn’t.
You can create a self signed certificate on your server. For any Unix based server you can use OpenSSL to generate it. The internet is filled with tutorials galore. Most of them will have information on how to install with your web server of choice (for example apache or nginx).
Once you’re done, go back to the CloudFlare settings for your domain and change the option to Full SSL.
You’re pretty secure if you ask me. The vulnerable points in the system are the CloudFlare server and your server. Ideally, CloudFlare protects the identity of the server ensuring all requests go through their servers which are meant to protect you from attacks. Lets assume CloudFlare’s private key isn’t compromised. This means the only way to decrypt requests to your server is by getting the private key off your machine and listening into requests to your server.
Another potential area of concern is the fact that the CloudFlare to server communication is prone to a Man in the middle attack since CloudFlare doesn’t verify the signature from from the server for free accounts. For this, you would need to move to a CloudFlare Pro account which at $20/month (as of the writing of this post) is significantly more expensive than purchasing your own SSL certificate end to end. Of course, we don’t use CloudFlare just for the free SSL but the CDN and crowd sourced security it provides. This is hands down the most serious vulnerability in the system I can spot.
You’re better off than where you started and all this for free! Want more? Shell out money to a CA to get your very own certificate :)
*[CA]: Certificate Authority
]]>I am concerned because they seem to not be getting their basics right. Most startup schools and accelerators tell you things that usually go wrong. Repeatedly. Yet there are other things they don’t tell you. Things they’d think are common sense.
Sometimes it is hard to spot these from the outside but if you do see these signs, you better have taken your ERT training seriously because the building is on fire and you better be prepared to evacuate people.
Everyone wants to be the “next big thing”. Everyone thinks they are the next Microsoft, Google, Facebook or Flipkart. Once in a while they are right. Usually they are wrong. Wanting to be great isn’t a problem. Walking around thinking you are is. If you’re Tony Stark and can pull it off, that’s good for you. Most can’t.
Some people want to do good work every day. It makes them happy. Success, for them, is a by product of doing great work.
Others want to be successful. The journey, to them, is meaningless and the destination is all that counts. Such people do not want to take the time to hone their craft. Usually such folks are impatient and won’t give the time it takes for the business to grow.
Take it from everyone who’s ever built anything. It’s not perfect. It’s never going to be. Unless you’re quite loose with your definition of the word, perfection is near impossible to attain. Delaying your launch is going to be your company’s death sentence.
This is the exact opposite of the previous point.
If you have a Minimum Viable Product and your client want’s to pay you for it, it’s great! You can use the money to support the cost of additional features. This is the recommended way to go to market.
Some entrepreneurs however would like to build the product entirely on the client’s cost. Believe it or not, this happens more often than one would think. How is this possible you ask?
Ethics are important. It is important how organisations treat_ _people.
When it comes to their clients, I believe they deserve to know what they are actually buying. This means being truthful about what features your product has. Claiming your product does things which you’ve not even started developing is a clear no-no.
When it comes to their own people, clear communication is important. Respecting them is key.
Hiring is key. You can have the greatest product the world has ever seen but if you hire incorrectly, you’ll have problems. This is a well known fact.
What most people don’t think of is that if you hire people and treat them as employees and not as stakeholders, they are going to treat work as a “job” and not as “their baby”. If I were you, I’d like my team to have that sense of ownership. It’s not easy but it’s not impossible either. Responsibility is to be given (stop trying to control everything; stop being a control freak) and responsibility is to be accepted. It’s a two way street. But it starts with your show of faith. It starts with you wanting to let people into the bubble you’ve built around yourself.
Not taking client feedback is almost always a silly move to play. Most people don’t do that.
What a few small organizations will miss out on is not listening to feedback from people they’ve hired build their product. First off, your team needs to be convinced about the product they are building. If they have feedback about the way things are being done, you should take it seriously. You’ve brought them in because you trust them and what they can do. So when they have things to say you better listen to do even if it doesn’t agree with your view of the world.
If the team always agree with one another, I’d consider it a orange flag. If they ostracise anyone who disagrees with them, that’s a definite red flag. Any team who discourages healthy discussions instead of understanding different view points and coming to a consensus is one to stay away from.
If you’re in discussion with such a team (as an insider or an outsider) and your points are something they disagree with but aren’t able to refute, you might see them not conceding and admitting you’re correct. Which brings us to the next point.
If you’re building something, you keep bottles of Kool Aid handy to give out to your users/clients. You must not dip into your own stash and take a sip or worse, gulp down entire bottles daily. Keeping your horizon indicator in check is key and large quantities of Kool Aid don’t help. They provide a self affirming view of the world which could be your startup’s death sentence. When it comes down to it, you should be ready to pivot for which you shouldn’t have blinders on.
Fail Fast. Learn. Move on. If you can’t do that, you’re doing it wrong.
I realise it sounds cliche but people like showing their world their perfectness. If you haven’t already, I recommend watching The Myth of the Genius Programmer from Google I/O 2009. It’s a real eye opener. Especially the part where they mention why failed spike branches shouldn’t be deleted (from 35:51 to 36:35).
For most people these days making it big is important. This could either be in terms of huge revenues or a 100+M dollar buy out. For most, having a steady NOI is just not good enough. It’s not good enough because it’s not what they had dreamt of. That Ferrari in the drive way and early retirement at 28.
Sounds outlandish but the Hollywood romanticisation of how businesses become worth billions has lead to people wanting to live the dream. Well son, this is reality. It’s hard work.
These are just some of the things which I believe everyone should think about. They seem obvious but heck, hindsight is always 20/20. It is quite easy to make some of these mistakes. Chances are you’ve seen someone around you exhibit these traits.
I highly recommend regular retrospectives with your team. More importantly, I recommend regular self retrospectives. Glass houses and all.
]]>Net Neutrality is principle that all data on the internet needs to be treated equally by service providers and governments. This means for example your YouTube videos should stream at the same speed your Vimeo videos*all other conditions being equal.
Would you want your internet service provider to slow down YouTube so as to make you move to one of its competitors?
This is a very brief description. At the end of the post are some resources to help you understand what Net Neutrality is if you’re new to this discussion.
Do you love your access to the internet? In this day and age, I view the Internet as a medium to impart information to those who didn’t have access to resources earlier. Over the past 15 years the introduction and mass penetration of smartphones in India is probably one of the greatest things that to have happened to our country. Have you ever gone driving between states with a map in your bag, gotten lost and spent time on the side of the road getting conflicting information about where you should go? I have. It sucks. With easy access to online maps, that’s rarely a concern these days :) This isn’t the only use case I can quote. I’m sure everyone has one. Yet, we undermine the significant contribution the internet has had in our lives (and I’m not talking about an endless stream of cat photographs on Facebook :))
You walk over to your local vegetable market and ask for 1kg of tomatoes. The vendor says it’s Rs. 10/kg if you’re making a salad and Rs. 30/kg if you’re making a curry. Would you be fine with this? Why does your vendor care what you’re doing with it?
Does your electricity company ask you for different rates for charging cell phones/laptops vs using TVs vs using ACs? It’s electricity. A unit is charged the same way irrespective of what you do with it.
Why should ISPs charge you different rates?
No. They don’t.
With Net Neutrality
Regular data - normal speed
‘Special’ data - normal speed
Without Net Neutrality when you pay them extra
Regular data - slower speed
‘Special’ data - normal speed
Without Net Neutrality when you don’t pay them extra
Regular data - slower speed
‘Special’ data - slower speed
This is, as John Oliver puts it, a classical mob shake down. Video is at the end of the post
Note: ‘Special’ Data refers to whatever they want to charge you extra for. Might be something like video streaming or VOIP.
Just so we’re clear, this is something I’ve heard in the Indian context. The services in question are VOIP services (such as Skype and Viber) and messaging services (mainly WhatsApp).
Lets take the example of WhatsApp. Telecos seem to be losing out on revenue for SMS services. At 30p/SMS, SMS is the most expensive (if not one of the most) form of communication used by the masses I can think of. The extremely high cost is precisely the reason why services like WhatsApp and Hike were able to come into the market.
Not being able to compete in a sector is no reason to stifle it using your monopoly as a mobile internet provider.
Lets say all dairies in your country sold cheese at insane rates. What if you found a way to produce (arguably?) a better quality cheese product for a lower price than the one dairies sell their’s at despite having to buy milk from them because you don’t have a source. Would I be fine with them increasing milk rates so you couldn’t afford to make the cheese cheaper any more? I wouldn’t. That’s why I don’t think charging extra for online messaging or VOIP services is acceptable.
No. It won’t. For two reasons mainly.
Each telecom provider has tens if not hundreds of mobile users around the country. If your aim is to make them feel the loss of revenue, you’d have to have their revenue to go down at least a few percentage points. This means making 100,000 to a million (10,000,000) users switching service. If you had that kind of support, you probably have a better chance getting the government to take notice and do something.
Sure, if there was an provider which stood by Net Neutrality, chances are that most of us would move to use their service. Not that it would help, but it’s a matter of principle. Still, that doesn’t fix the problem at hand.
Multiple major players in the market have considered violating neutrality in some form over the past few years. Even ones who haven’t would probably do so once they see their competitors increasing their earnings.
We’re not. Not to sound gloomy and all but the battle is far from won. There is affirmative action still required to ensure Internet in India remains a level playing field. We need to ensure people with vested interests do not take advantage of the system and this can only happen through awareness. There are multiple issues plaguing internet access in India not the least of which are privacy and (lawful/unlawful; a debate in itself) censorship.
Sure you are. You should never underestimate the power of one person in a democracy. Learn more about this issue. Become an advocate. Make others around you aware why this is a concern. Awareness and will to stand up against this is the only way to go forward.
There are tons of great YouTube videos out there but there are 2 in general I recommend that I’ve come across over the past year or so. Each of these works on different sets of people.
CGP Grey hits it out of the park with amazing visualisations explaining what is Net Neutrality. It is a bit fast paced as all his videos are so you might want to rewind and hear stuff again if it’s too fast for you.
John Oliver does what he does best. Makes fun off things and explains important issues.
“please don’t eat my baaaby..”
]]>Once you’re done, restart the application that was attempting to use your web camera :)
]]>