1. Chef Basic Training (English version) RYUZEE.COM

2. About RYUZEE.COM ✤ Consulting Service Provider ✤ Agile Development / DevOps / Cloud Computing ✤ http://www.ryuzee.com

3. What is DevOps?

4. Current Challenges ✤ Business changes faster ✤ IT becomes a key element for business ✤ Delivering good software to customer is critical for business ✤ However, the speed of delivery is too slow and error prone (especially in traditional or large companies…) ✤ This causes the lost of chance and money in business ✤ Or you have your own issues ✤ IT often becomes Bottleneck

5. Muda Always 7% Some.mes 16% Never 45% O2en 13% Rarely 19% ✤ 64 % of all features were rarely or never used (Excerpt from The Standish Group Chaos report 2002). ✤ You can imagine it by taking your computer s builtin softwares usage into consideration…

6. Seven types of Waste ✤ Waste of over-production ✤ Waste of waiting ✤ Waste of transportation ✤ Waste of processing ✤ Waste of inventory ✤ Waste of motion ✤ Waste of making defects

7. It s difficult to know right things or future in advance. It s important to have a capability to catch up with fast changes.

8. Building fast feedback cycle This is not only for Learn Bu il d s a e M e r u

9. Conflict between Dev and Ops ✤ Differences of mission and responsibility (Who decides them…? Reasonable?) ✤ It s not my business (Says who?) ✤ Silo ✤ It creates overhead and slows down your business result

10. ✤ Ops who think like devs. ✤ Devs who think like ops.

11. Werner Vogels, CTO, amazon.com You build it, You run it

12. DevOps intends… ✤ DevOps intends to achieve business results, to enhance business agility and to avoid or reduce business risks by leveraging culture and tool.

13. Five Aspects ✤ Culture ✤ Lean ✤ Automation ✤ Measurement ✤ Sharing

14. Why do we need Infrastructure as Code?

15. Challenges in manual provisioning ✤ It takes longer time when the number of target servers increases ✤ Procedure documents or check lists is not maintained and kept updated ✤ Document driven manual operation causes mis-operation ✤ Documents can not be re-used across organisation

16. Imagine it…

17. Benefits of Infrastructure as Code ✤ The duration of provisioning is almost tranquil ✤ Code equals to Procedure Document . Only you need to do is to keep the code updated ✤ Code runs as it wrote. Same code bring us a same server ✤ Code can be tested continuously by using CI tools ✤ High reusability

18. How to automate infrastructure

19. There are several ways to automate ✤ Shell Script ✤ Capistrano or other deploy tools ✤ Provisioning tools such as Chef / Puppet / Ansible

20. Shell Script #!/bin/sh yum install -y httpd httpd-devel php phpmbstring php-pdo php-mysql mysql-server /sbin/chkconfig --level 2345 httpd on /sbin/chkconfig --level 2345 mysqld on /etc/rc.d/init.d/mysqld start /etc/rc.d/init.d/httpd start ✤ Simplest way ✤ However long script that includes conditional statements can not be maintained easily

21. Deploy Tool (Capistrano) task :install_amp, roles => :web do ✤ Capistrano or other deploy tools intend to be used for application deployment ✤ Of course, It s not impossible to automate infrastructure provisioning by deploy tools… run <<-CMD sudo yum install -y httpd httpd-devel php phpmbstring php-pdo php-mysql mysql-server && sudo /sbin/chkconfig --level 2345 httpd on && sudo /sbin/chkconfig --level 2345 mysqld on && sudo /etc/rc.d/init.d/mysqld start && sudo /etc/rc.d/init.d/httpd start CMD end

22. Provisioning Tool (Chef) %w{httpd httpd-devel php php-mbstring php-pdo php-mysql mysql-server}.each do ¦p¦ package p do action :install end end service "httpd" do action [:enable, :restart] supports :status => true, :start => true, :stop => true, :restart => true end service "mysqld" do action [:enable, :restart] supports :status => true, :start => true, :stop => true, :restart => true end

23. Several Provisioning Tools ✤ Chef Ansible Puppet DSL (Ruby based) DSL DSL Client / Server (Agent) Agentless Client / Server (Agent) Lots of related tools such as knife, berkshelf, foodcritic… Few management tools Already old… DSL(Ruby based) must be learned Simple. a few to learn Unique DSL must be learned There are many server management tools that are written in Ruby. It s better to learn Ruby although you are an infrastructure engineer.

24. Chef's Architecture

25. Architecture ✤ Basically the architecture is Client / Server model ✤ Every information is stored in Chef Server and all nodes will access to Chef Server (From clients to Chef Server) to retrieve cookbooks, various attributes and so on

26. Basic Terminology #1 ✤ Chef Server => The Chef Server acts as a hub for configuration data. The Chef Server stores cookbooks, the policies that are applied to nodes, and metadata that describes each registered node that is being managed by the Chef Client ✤ Nodes => A node is any machine̶physical, virtual, cloud, network device, etc.̶that is under management by Chef. Chef Client must be installed ✤ Chef Client => Tool to be installed into Nodes. It can be run as Service (daemon) or command line tool

27. Basic Terminology #2 ✤ Cookbook => A cookbook is the fundamental unit of configuration and policy distribution. A cookbook defines a scenario and contains everything that is required to support that scenario. Cookbook contains Recipe, Attributes, Files, Templates and custom extensions ✤ Recipe => DSL (Ruby based) code to install or configure target nodes. A cookbook can contain multiple recipes

28. NOTE: Chef Solo ✤ Chef also had a NON client / server mode named Chef Solo ✤ However, Chef Solo is now deprecated ✤ If you want to run Chef without Server, you are going to use Chef Local Mode (via knife-zero) ✤ Many web resources still pointed out Chef Solo. However, you need to remember the above.

29. Hands-on Environment

30. Environment Overview Hostname: development IP Address: 192.168.33.10 OS: Ubuntu 14.04 Chef-DK / Docker was installed vi / vim / emacs are available Login to this virtual machine Development Hostname: node01 IP Address: 192.168.33.200 OS: Ubuntu 14.04 Chef Client was installed Machine to be provisioned Node01 Vagrant: Manage lightweight, reproducible, and portable development environments by HashiCorp VirtualBox: Open source virtualisation tool provided by Oracle Your Laptop

31. You can download VirtualBox from https://www.virtualbox.org/

32. You can download Vagrant from https://www.vagrantup.com/ If you already installed Vagrant, please update it to 1.8+

33. Vagrant Basics ✤ Vagrant is one of the most popular open source tool to manage development environments provided by HashiCorp ✤ Vagrant can run VirtualBox virtual machine, Docker machine, Azure virtual machine and so on. Vagrant wraps the deferences ✤ Vagrantfile is the definition of the environment. Same Vagrantfile produces same virtual environment. Thus your team members are able to obtain the same development environment. It could be a quite plus when you are developing something

34. Vagrantfile Example Vagrant.configure(2) do ¦config¦ config.vm.define :development do ¦development¦ development.vm.box = 'ubuntu-14.04.4-chef-training-development-kit' development.vm.hostname = 'development' development.vm.network 'private_network', ip: '192.168.33.10' end config.vm.define :node01 do ¦node01¦ node01.vm.box = 'ubuntu-14.04.4-chef-training-node' node01.vm.hostname = 'node01' node01.vm.network 'private_network', ip: '192.168.33.200' end end You can download the script from http://bit.ly/224TbdH

35. Vagrant basic commands (built-in) # boot virtual machines vagrant up # stop machines vagrant halt [machine name] # boot specified machine vagrant up development # dispose machines vagrant destroy [machine name] # login to the specific machine vagrant ssh development # add box as a template vagrant box add box_name box_url # reboot machines vagrant reload [machine name] # install plugin vagrant plugin install plugin_name

36. Add boxes ✤ Add boxes (from terminal or command prompt) vagrant box add ubuntu-14.04.4-chef-training-development-kit http://bit.ly/1W4FWtV vagrant box add ubuntu-14.04.4-chef-training-node http://bit.ly/1PQjMEl

37. All preparations finished? Then… vagrant up ✤ Move to the directory that contains Vagrantfile, and then execute the command above (Terminal or Command Prompt) ✤ It will launch 2 virtual machines ✤ If it fails, please check logs or stderr

38. Writing first cookbook

39. Automate to install nginx ✤ nginx [engine x] is an HTTP and reverse proxy server. The performance is better than Apache HTTP Server ✤ Now we are going to automate to install nginx via Chef ✤ Please login to virtual machine named development by typing vagrant ssh development in terminal (OS X) or command prompt (Windows)

40. Only For Windows User ✤ Unfortunately Vagrant on Windows does not provide vagrant ssh functionality. ✤ Thus, if you want to login to virtual machine, open preferred ssh client such as Teraterm or Putty and then ssh access to 192.168.33.10 with username: vagrant and password vagrant

41. Login to development environment ✤ You can look at the message as follows ✤ Now you are in the Linux(Ubuntu) virtual machine ✤ Username is vagrant and the current path is /home/vagrant Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 4.2.0-27-generic x86_64) * Documentation: https://help.ubuntu.com/ Last login: Fri Apr 1 20:54:29 2016 from 10.0.2.2 vagrant@development: $

42. Git setup ✤ Before staring hands-on, setup your git account as follows git config --global user.name Sushi Taro git config --global user.email taro@example.com

43. Create Repository ✤ At first, create repository to store cookbooks, node information and so on chef generate repo chef-repo cd chef-repo

44. chef command shows… Installing Cookbook Gems: Compiling Cookbooks... Recipe: code_generator::repo * directory[/home/vagrant/chef-repo] action create (up to date) * template[/home/vagrant/chef-repo/LICENSE] action create_if_missing (up to date) * cookbook_file[/home/vagrant/chef-repo/.chef-repo.txt] action create_if_missing (up to date) * cookbook_file[/home/vagrant/chef-repo/README.md] action create_if_missing (up to date) (snip) Recipe: code_generator::repo * cookbook_file[/home/vagrant/chef-repo/cookbooks/README.md] action create_if_missing (up to date) * execute[initialize-git] action run - execute git init . * template[/home/vagrant/chef-repo/.gitignore] action create_if_missing - create new file /home/vagrant/chef-repo/.gitignore - update content in file /home/vagrant/chef-repo/.gitignore from none to 3523c4 (diff output suppressed by config)

45. Directory Tree ✤ tree -L 2 shows the directory structure and files. ✤ You can see cookbooks directory that stores chef cookbook. vagrant@development: /chef-repo$ tree -L 2 . ¦-- chefignore ¦-- cookbooks ¦ ¦-- example ¦ `-- README.md ¦-- data_bags ¦ ¦-- example ¦ `-- README.md ¦-- environments ¦ ¦-- example.json ¦ `-- README.md ¦-- LICENSE ¦-- README.md `-- roles ¦-- example.json `-- README.md 6 directories, 9 files

46. Create nginx cookbook from template ✤ run knife cookbook create nginx -o ./cookbooks/ command in the current directory. It creates base files that consist of the cookbook ✤ Confirm the structure following to the right screenshot vagrant@development: /chef-repo$ tree -F 2 ./cookbooks/ nginx/ 2 [error opening dir] ./cookbooks/nginx/ ¦-- attributes/ ¦-- CHANGELOG.md ¦-- definitions/ ¦-- files/ ¦ `-- default/ ¦-- libraries/ ¦-- metadata.rb ¦-- providers/ ¦-- README.md ¦-- recipes/ ¦ `-- default.rb ¦-- resources/ `-- templates/ `-- default/ 10 directories, 4 files

47. Implement cookbook ✤ ✤ edit ./cookbooks/nginx/recipes/ default.rb and input text indicated at the right This intends to install nginx package, enable nginx service and run service package 'nginx' do action :install end service 'nginx' do action [ :enable, :start ] end

48. Setup Chef Client knife zero bootstrap 192.168.33.200 -x vagrant --sudo --ssh-password vagrant ✤ This command will install Chef Client in target node and create configuration file in source environment ✤ After this command, you will find some new directories such as nodes and clients in /home/vagrant/chef-repo/ directory

49. Confirm the target node is registered knife node list -z ✤ ✤ node01 will be shown If you are handling several environments, all nodes will be displayed

50. Set run_list knife node run_list add node01 'recipe[nginx]' -z ✤ This command means that we are going to apply the default recipe in nginx cookbook to the target node named node01 ✤ This command will update the configuration file nodes/ node01.json . Please confirm bottom of that file. ✤ You can specify multiple recipes at the same time

51. Take a snapshot of Virtual Machine vagrant snapshot save node01 node01_001 ✤ Run the command above in your host environment (NOT guest) ✤ vagrant snapshot is a built-in command to take snapshots and restore them ✤ To shorten the waiting duration, it s better to leverage various kind of tools

52. Apply changes to target server knife zero converge 'name:node01' -x vagrant --sudo -a knife_zero.host --ssh-password vagrant ✤ This command will apply changes to node01. In Chef World, we usually say that node will be converged into specific state ✤ Open web browser and access to http://192.168.33.200

54. Are you OK? ✤ If it works well, add and commit files via git as follows git add . git commit -m initial commit ✤ If your cookbook does not work, check stdout and try again

55. Template #1 ✤ ✤ Chef can generate and provision files by using template functionality Try to change index.html for nginx. Create a new file named index.html.erb in cookbooks/nginx/templates/ default/ with the content displayed in the right side <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> </head> <body> <h1>Welcome to Chef Training</h1> <p><%= node.name %></p> </body> </html>

56. Template #2 ✤ Now, update the recipe ✤ Add highlighted section to cookbooks/nginx/recipes/ default.rb ✤ Before converge, it s better to run vagrant snapshot restore node01 node01_001 at the host machine to restore VM package 'nginx' do action :install end template 'index.html' do path '/usr/share/nginx/html/index.html' owner root group root mode 0644 end service 'nginx' do action [ :enable, :start ] end

57. Apply changes to target server (Again) knife zero converge 'name:node01' -x vagrant --sudo -a knife_zero.host --ssh-password vagrant ✤ This command will apply changes to node01. In Chef World, we usually say that node will be converged into specific state ✤ Open web browser and access to http://192.168.33.200

59. Resource ✤ At this point, you tried package , service , template in your first cookbook. These keywords are called as resource ✤ Chef has lots of built-in resources (See next page) ✤ You are going to write your own cookbook by leveraging various resources

60. Resources (Excerpt…) package user powershell_script ifconfig template group ruby_block http_request service remote_file cron link file execute git log directory script mount chai

61. Basic Terminology #3 & need to learn ✤ Attribute => Attribute(s) are variables that can be used when provisioning. For example, when you create php environment, you may want to set several variables in php.ini. You can set these values when provisions. ✤ Role => You can define Roles as you like. Typically, web server role, db server role, monitoring server role and so on. By using role, you can build specific infrastructure by only selecting role(s) for setting several recipes ✤ Environment => You can have several environment such as development, staging, production that have different environmental values(attributes)

62. Test automation

63. Why automated tests matter? ✤ Reduce risks ✤ Reduce repetitive manual processes ✤ Generate deployable software at any time and at any place ✤ Enable better project visibility ✤ Establish greater confidence in the software product from the development team

64. Test Kitchen ✤ Test Kitchen is a test harness tool to execute your configured code on one or more platforms in isolation ✤ See more details at http://kitchen.ci ✤ Test Kitchen launches isolated environment (using Vagrant, Docker, AWS, Azure…), apply specified recipes in the cookbook, verify results ✤ It supports many testing frameworks including Bats, shUnit2, RSpec, Serverspec

65. Visit http://serverspec.org/

66. Preparation ✤ Test Kitchen provides test generator. Run command as follows cd /home/vagrant/chef-repo/cookbooks/nginx kitchen init -D kitchen-docker ✤ .kitchen.yml, chefignore, test/integration/default must be created mkdir -p test/integration/default/serverspec/localhost

67. Write Tests require spec_helper describe package("nginx") do it { should be_installed } end describe service("nginx") do it { should be_enabled } it { should be_running } end test/integration/default/serverspec/ localhost/default_spec.rb test/integration/default/serverspec/ spec_helper.rb describe port(80) do it { should be_listening } end describe file( /usr/share/nginx/html/index.html ) do it { should be_file } end require serverspec set :backend, :exec

68. What are testing for? require spec_helper describe package("nginx") do it { should be_installed } end ✤ nginx package should be installed ✤ nginx service should be running describe service("nginx") do it { should be_enabled } it { should be_running } end describe port(80) do it { should be_listening } end describe file( /usr/share/nginx/html/index.html ) do it { should be_file } end ✤ nginx service should run after boot ✤ nginx should be listening on TCP/80 ✤ index.html should exist

69. Edit configuration and start verification --driver: name: docker provisioner: name: chef_solo platforms: - name: ubuntu-14.04 suites: - name: default run_list: - recipe[nginx::default] attributes: ✤ Edit .kitchen.yml as noted in the left ✤ It means that test will run Ubuntu14 Docker machine ✤ Execute kitchen test and then verification starts!! ✤ Verification will take a few minutes

70. You can see the verification result

71. Run Tests with Jenkins ✤ It s possible to run tests with Jenkins ✤ If you are interested in implement it, please try it later ✤ Install JDK8 (NOT JDK7), Jenkins into development machine as follows sudo add-apt-repository ppa:openjdk-r/ppa sudo apt-get update sudo apt-get install openjdk-8-jdk wget http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_1.642.4_all.deb sudo dpkg -i jenkins_1.642.4_all.deb

72. How to write good cookbooks

73. Community Cookbooks ✤ Chef has a huge eco-system. Many cookbooks (see the table below) were released by chef community. Visit https:// supermarket.chef.io/ mysql nginx apache2 postgresql java git apt yum php build-essential nodejs mongodb ntp jenkins database python docker tomcat rabbitmq elasticsearch

74. Berkshelf : Dependency manager ✤ Especially in community cookbook, there might be some dependencies on other cookbooks. ✤ To resolve this challenge, you can use Berkshelf. See http:// berkshelf.com/ ✤ It is similar to other language package manager such as composer(PHP), bundler(Ruby) and npm(Nodejs)

75. Keep the code Clean ✤ Cookbook is equal to the procedure document. So readability and maintainability is important (Also continuous integration matters) ✤ For example, Foodcritic verifies your cookbooks (static analysis) like rubocop ✤ Keep cookbook small that meet the Single Responsibility principle

76. Questions?