Snap CI enables software teams to do Continuous Delivery (CD). When practicing CD, the goal is to automate the deployment process and build software in such a way that it can be deployed to production any time. As a deployment tool, Snap CI cannot have downtime. If it did, our users would not be able to deploy their own software. We had to change Snap CI’s architecture to ensure zero-downtime and we chose to do blue-green deployments to achieve it. In this approach, we had to maintain two instances of our system: one active instance, and one inactive instance. Based on our experiences, we will share some tricks of the trade from the numerous challenges we faced such as: making the application aware of whether it was active or inactive, handling data migrations, and babysitting long-running jobs. These are the slides from Akshay Karle and Fernando Junior's presentation on Agile Brazil 2015.

1. Continuously serving the developer
community with
Continuous Integration and Delivery
Akshay Karle
Fernando Júnior

2. “How long would it take your
organization to deploy a change that
involves just one single line of code?”
Mary and Tom Poppendieck

3. “(…) build software in such a way that
the software can be released to
production at any time.”
Martin Fowler

4. CONTINUOUS INTEGRATION

5. CONTINUOUS DELIVERY

6. CONTINUOUS DELIVERY
▫︎ Continuous Integration
▫︎ Automated tasks
▫︎ Repeatable/reliable process

7. CONTINUOUS DELIVERY
frequent, reliable releases
of high quality valuable
software

8. THERE’S MORE…

9. CONTINUOUS DELIVERY

10. CONTINUOUS DELIVERY

11. DEPLOYING YOUR APPLICATION
▫︎ Database migrations
▫︎ Infrastructure update
▫︎ Restarting services

12. ZERO DOWNTIME DEPLOYMENTS
Deployment process of your application
has got to be transparent for end users

13. WHO WE ARE?
Developer,
ThoughtWorks
@nandopaf
fernando-alves
nand0paf
Developer,
ThoughtWorks
@akshay_karle
akshaykarle
akshay_ka

14. WHAT WE DO?
▫︎Continuous Integration and Delivery for repositories on
GitHub
▫︎Cloud Hosted

15. SNAP INSIDE-OUT

16. LIFECYCLE OF A BUILD
head
repository
commit

17. LIFECYCLE OF A BUILD
head
repository
commit

18. Babysitters
LIFECYCLE OF A BUILD
head
repository
commit
DATABASE

19. LIFE CYCLE
▫︎ Prepare the container
▫︎ Starts database
▫︎ Sets PATH for languages
▫︎ Download artifacts
▫︎Git clone
▫︎Runs each pipeline stage
▫︎Upload artifacts
DATABASE

20. Babysitters
LIFECYCLE OF A BUILD
head
repository
commit
DATABASE

21. ARCHITECTURE OVERVIEW
L
B
Database
web server

22. ARCHITECTURE OVERVIEW
▫︎ Rails app fronted by
nginx
▫︎ Receives the hooks from
GitHub
Database
web server

23. ARCHITECTURE OVERVIEW
L
B
Build Server
web server
Database

24. ARCHITECTURE OVERVIEW
▫︎ Background jobs
▫︎ Babysitters
▫︎ Build Queue
▫︎ Artifacts
▫︎ OpenVZ Containers
▫︎ Virtual machines where the
pipeline runs
Build Server

25. ARCHITECTURE OVERVIEW
L
B
Build Server
web server
Database

26. OUR DEPLOYMENT
▫︎ Do have automated scripts
▫︎ Deployment pipeline
▫︎ 1-click deploy
▫︎ Sort of…
Babysitters
DATABASE
L
B
Build Server
Database
web server

27. OUR DEPLOYMENT
▫︎ Wait for all builds to finish
▫︎ Put app on maintenance
mode
▫︎ No new requests picked up
▫︎ Deploy and wait for
migrations
Babysitters
DATABASE
L
B
Database
VZHOSTBuild Server
web server

28. ZERO DOWNTIME
DEPLOYMENTS

29. BLUE-GREEN DEPLOYMENT
Web Server App Server Database
App ServerApp ServerWeb Server Database

30. BLUE-GREEN DEPLOYMENTS FOR SNAP
VZHOSTBuild Server
web server
VZHOSTBuild Server
web server
L
B
DATABASE

31. CHALLENGES IN SNAP
Long running builds
Running multiple versions of your app at the
same time
Database migrations

32. LONG RUNNING BUILDS
▫︎ Builds should continue
to run
▫︎ Artifacts should continue
to be served

33. COLOR AWARE BABYSITTERS
Build Server
Green Stack
Build Server
Blue Stack
web server

34. BUILD LIFECYCLE
head
repository
commit
DATABASEDATABASE
L
B

35. THE SWITCH
L
B

36. THE SWITCH
L
B

37. CHALLENGES IN SNAP
Long running builds
Running multiple versions of your app at the
same time
Database migrations

38. FEATURE TOGGLES
Hide unfinished UI elements
Control backend behaviour
Test with feature toggles
Avoid multi-component feature toggles
http://martinfowler.com/bliki/FeatureToggle.html

39. FRONTEND
<% if feature_enabled?(:parallel_stage) %>
<!-- show new html -->
<% else %>
<!-- show old html -->
<% end %>

40. BACKEND
if feature_enabled?(:parallel_stage)
# new logic
else
# old logic
end

41. TESTING WITH FEATURE TOGGLES
describe "multiple jobs" do
describe "feature enabled" do
before(:each) do
with_feature_enabled(:parallel_stage)
end
it 'should not show the job tabs when there is only one job' do
end
end
describe "feature disabled" do
before(:each) do
with_feature_disabled(:parallel_stage)
end
it 'should not show the job tabs but should show the logs' do
end
end
end

42. CHALLENGES IN SNAP
Long running builds
Running multiple versions of your app at the
same time
Database migrations

43. MIGRATION OF DATA
Consider existing schema as well as data
Incremental changes
Rollbacks

44. INTRODUCING JOBS
jobs
id
…

45. INTRODUCING JOBS
stages
id
started_at
completed_at
result
…
jobs
id
…

46. POPULATE ALL EXISTING STAGES
stages
id
started_at
completed_at
result
…
jobs
id
stage_id
…

47. COPY ATTRIBUTES TO JOB
stages
…
started_at
completed_at
result
…
jobs
…
started_at
completed_at
result
…

48. SWITCH THE APPLICATION TO START USING
THE JOB MODEL
# After switch
class Stage
def result
results = jobs.collect { |job| job.result }
return :failed if results.any?(:failed)
:passed
end
end
# Before switch
class Stage
attr_reader :result
end
# in transition
class Stage
def result
if feature_enabled?(:parallel_stage)
results = jobs.collect { |job| job.result }
return :failed if results.any?(:failed)
:passed
else
result
end
end
end

49. REMOVE UNUSED ATTRIBUTES
stages
id
started_at
completed_at
result
…
jobs
…
started_at
completed_at
result
…

50. LESSONS LEARNT

51. LESSONS LEARNT
Zero downtime is not easy
Having reliability, automation, frequent releases helped
Watch out your data
Things will go wrong, but we keep learning and keep improving

52. THANK YOU
Questions?
https://snap-ci.com