The Practice of System and Network Administration: The DevOps and other Best Practices for Enterprise IT (Volume 1) (3rd Edition)

DescriptionWith 28 new chapters, the third edition of The Practice of System and Network Administration innovates yet again! Revised with thousands of updates and clarifications based on reader feedback, this new edition also incorporates DevOps strategies even for non-DevOps environments.Whether you use Linux, Unix, or Windows, this new edition describes the essential practices previously handed down only from mentor to protégé. This wonderfully lucid, often funny cornucopia of information introduces beginners to advanced frameworks valuable for their entire career, yet is structured to help even experts through difficult projects.Other books tell you what commands to type. This book teaches you the cross-platform strategies that are timeless!DevOps techniques: Apply DevOps principles to enterprise IT infrastructure, even in environments without developersGame-changing strategies: New ways to deliver results faster with less stressFleet management: A comprehensive guide to managing your fleet of desktops, laptops, servers and mobile devicesService management: How to design, launch, upgrade and migrate servicesMeasurable improvement: Assess your operational effectiveness; a forty-page, pain-free assessment system you can start using today to raise the quality of all servicesDesign guides: Best practices for networks, data centers, email, storage, monitoring, backups and more Management skills: Organization design, communication, negotiation, ethics, hiring and firing, and moreHave you ever had any of these problems?Have you been surprised to discover your backup tapes are blank?Ever spent a year launching a new service only to be told the users hate it?Do you have more incoming support requests than you can handle?Do you spend more time fixing problems than building the next awesome thing?Have you suffered from a botched migration of thousands of users to a new service?Does your company rely on a computer that, if it died, can’t be rebuilt?Is your network a fragile mess that breaks any time you try to improve it?Is there a periodic “hell month” that happens twice a year? Twelve times a year?Do you find out about problems when your users call you to complain?Does your corporate “Change Review Board” terrify you?Does each division of your company have their own broken way of doing things?Do you fear that automation will replace you, or break more than it fixes?Are you underpaid and overworked?No vague “management speak” or empty platitudes. This comprehensive guide provides real solutions that prevent these problems and more!Table of ContentsPreface xxxixAcknowledgments xlviiAbout the Authors liPart I: Game-Changing Strategies 1Chapter 1: Climbing Out of the Hole 3Organizing WIP 5Eliminating Time Sinkholes 12DevOps 16DevOps Without Devs 16Bottlenecks 18Getting Started 20Summary 21Exercises 22Chapter 2: The Small Batches Principle 23The Carpenter Analogy 23Fixing Hell Month 24Improving Emergency Failovers 26Launching Early and Often 29Summary 34Exercises 34Chapter 3: Pets and Cattle 37The Pets and Cattle Analogy 37Scaling 39Desktops as Cattle 40Server Hardware as Cattle 41Pets Store State 43Isolating State 44Generic Processes 47Moving Variations to the End 51Automation 53Summary 53Exercises 54Chapter 4: Infrastructure as Code 55Programmable Infrastructure 56Tracking Changes 57Benefits of Infrastructure as Code 59Principles of Infrastructure as Code 62Configuration Management Tools 63Example Infrastructure as Code Systems 67Bringing Infrastructure as Code to Your Organization 71Infrastructure as Code for Enhanced Collaboration 72Downsides to Infrastructure as Code 73Automation Myths 74Summary 75Exercises 76Part II: Workstation Fleet Management 77Chapter 5: Workstation Architecture 79Fungibility 80Hardware 82Operating System 82Network Configuration 84Accounts and Authorization 86Data Storage 89OS Updates 93Security 94Logging 97Summary 98Exercises 99Chapter 6: Workstation Hardware Strategies 101Physical Workstations 101Virtual Desktop Infrastructure 105Bring Your Own Device 110Summary 113Exercises 114Chapter 7: Workstation Software Life Cycle 117Life of a Machine 117OS Installation 120OS Configuration 120Updating the System Software and Applications 123Rolling Out Changes . . . Carefully 128Disposal 130Summary 134Exercises 135Chapter 8: OS Installation Strategies 137Consistency Is More Important Than Perfection 138Installation Strategies 142Test-Driven Configuration Development 147Automating in Steps 148When Not to Automate 152Vendor Support of OS Installation 152Should You Trust the Vendor’s Installation? 154Summary 154Exercises 155Chapter 9: Workstation Service Definition 157Basic Service Definition 157Refresh Cycles 161Tiered Support Levels 165Workstations as a Managed Service 168Summary 170Exercises 171Chapter 10: Workstation Fleet Logistics 173What Employees See 173What Employees Don’t See 174Configuration Management Database 183Small-Scale Fleet Logistics 186Summary 188Exercises 188Chapter 11: Workstation Standardization 191Involving Customers Early 192Releasing Early and Iterating 193Having a Transition Interval (Overlap) 193Ratcheting 194Setting a Cut-Off Date 195Adapting for Your Corporate Culture 195Leveraging the Path of Least Resistance 196Summary 198Exercises 199Chapter 12: Onboarding 201Making a Good First Impression 201IT Responsibilities 203Five Keys to Successful Onboarding 203Cadence Changes 212Case Studies 212Summary 216Exercises 217Part III: Servers 219Chapter 13: Server Hardware Strategies 221All Eggs in One Basket 222Beautiful Snowflakes 224Buy in Bulk, Allocate Fractions 228Grid Computing 235Blade Servers 237Cloud-Based Compute Services 238Server Appliances 241Hybrid Strategies 242Summary 243Exercises 244Chapter 14: Server Hardware Features 245Workstations Versus Servers 246Server Reliability 249Remotely Managing Servers 254Separate Administrative Networks 257Maintenance Contracts and Spare Parts 258Selecting Vendors with Server Experience 261Summary 263Exercises 263Chapter 15: Server Hardware Specifications 265Models and Product Lines 266Server Hardware Details 266Things to Leave Out 278Summary 278Exercises 279Part IV: Services 281Chapter 16: Service Requirements 283Services Make the Environment 284Starting with a Kick-Off Meeting 285Gathering Written Requirements 286Customer Requirements 288Scope, Schedule, and Resources 291Operational Requirements 292Open Architecture 298Summary 302Exercises 303Chapter 17: Service Planning and Engineering 305General Engineering Basics 306Simplicity 307Vendor-Certified Designs 308Dependency Engineering 309Decoupling Hostname from Service Name 313Support 315Summary 319Exercises 319Chapter 18: Service Resiliency and Performance Patterns 321Redundancy Design Patterns 322Performance and Scaling 326Summary 333Exercises 334Chapter 19: Service Launch: Fundamentals 335Planning for Problems 335The Six-Step Launch Process 336Launch Readiness Review 345Launch Calendar 348Common Launch Problems 349Summary 351Exercises 351Chapter 20: Service Launch: DevOps 353Continuous Integration and Deployment 354Minimum Viable Product 357Rapid Release with Packaged Software 359Cloning the Production Environment 362Example: DNS/DHCP Infrastructure Software 363Launch with Data Migration 366Controlling Self-Updating Software 369Summary 370Exercises 371Chapter 21: Service Conversions 373Minimizing Intrusiveness 374Layers Versus Pillars 376Vendor Support 377Communication 378Training 379Gradual Roll-Outs 379Flash-Cuts: Doing It All at Once 380Backout Plan 383Summary 385Exercises 385Chapter 22: Disaster Recovery and Data Integrity 387Risk Analysis 388Legal Obligations 389Damage Limitation 390Preparation 391Data Integrity 392Redundant Sites 393Security Disasters 394Media Relations 394Summary 395Exercises 395Part V: Infrastructure 397Chapter 23: Network Architecture 399Physical Versus Logical 399The OSI Model 400Wired Office Networks 402Wireless Office Networks 406Datacenter Networks 408WAN Strategies 413Routing 419Internet Access 420Corporate Standards 422Software-Defined Networks 425IPv6 426Summary 428Exercises 429Chapter 24: Network Operations 431Monitoring 431Management 432Documentation 437Support 440Summary 446Exercises 447Chapter 25: Datacenters Overview 449Build, Rent, or Outsource 450Requirements 452Summary 456Exercises 457Chapter 26: Running a Datacenter 459Capacity Management 459Life-Cycle Management 465Patch Cables 468Labeling 471Console Access 475Workbench 476Tools and Supplies 477Summary 480Exercises 481Part VI: Helpdesks and Support 483Chapter 27: Customer Support 485Having a Helpdesk 485Offering a Friendly Face 488Reflecting Corporate Culture 488Having Enough Staff 488Defining Scope of Support 490Specifying How to Get Help 493Defining Processes for Staff 493Establishing an Escalation Process 494Defining “Emergency” in Writing 495Supplying Request-Tracking Software 496Statistical Improvements 498After-Hours and 24/7 Coverage 499Better Advertising for the Helpdesk 500Different Helpdesks for Different Needs 501Summary 502Exercises 503Chapter 28: Handling an Incident Report 505Process Overview 506Phase A—Step 1: The Greeting 508Phase B: Problem Identification 509Phase C: Planning and Execution 515Phase D: Verification 518Perils of Skipping a Step 519Customer Care 521Summary 525Exercises 527Chapter 29: Debugging 529Understanding the Customer’s Problem 529Fixing the Cause, Not the Symptom 531Being Systematic 532Having the Right Tools 533End-to-End Understanding of the System 538Summary 540Exercises 540Chapter 30: Fixing Things Once 541Story: The Misconfigured Servers 541Avoiding Temporary Fixes 543Learn from Carpenters 545Automation 547Summary 549Exercises 550Chapter 31: Documentation 551What to Document 552A Simple Template for Getting Started 553Easy Sources for Documentation 554The Power of Checklists 556Wiki Systems 557Findability 559Roll-Out Issues 559A Content-Management System 560A Culture of Respect 561Taxonomy and Structure 561Additional Documentation Uses 562Off-Site Links 562Summary 563Exercises 564Part VII: Change Processes 565Chapter 32: Change Management 567Change Review Boards 568Process Overview 570Change Proposals 570Change Classifications 571Risk Discovery and Quantification 572Technical Planning 573Scheduling 574Communication 576Tiered Change Review Boards 578Change Freezes 579Team Change Management 581Starting with Git 583Summary 585Exercises 585Chapter 33: Server Upgrades 587The Upgrade Process 587Step 1: Develop a Service Checklist 588Step 2: Verify Software Compatibility 591Step 3: Develop Verification Tests 592Step 4: Choose an Upgrade Strategy 595Step 5: Write a Detailed Implementation Plan 598Step 6: Write a Backout Plan 600Step 7: Select a Maintenance Window 600Step 8: Announce the Upgrade 602Step 9: Execute the Tests 603Step 10: Lock Out Customers 604Step 11: Do the Upgrade with Someone 605Step 12: Test Your Work 605Step 13: If All Else Fails, Back Out 605Step 14: Restore Access to Customers 606Step 15: Communicate Completion/Backout 606Summary 608Exercises 610Chapter 34: Maintenance Windows 611Process Overview 612Getting Management Buy-In 613Scheduling Maintenance Windows 614Planning Maintenance Tasks 615Selecting a Flight Director 616Managing Change Proposals 617Developing the Master Plan 620Disabling Access 621Ensuring Mechanics and Coordination 622Change Completion Deadlines 628Comprehensive System Testing 628Post-maintenance Communication 630Reenabling Remote Access 631Be Visible the Next Morning 631Postmortem 631Mentoring a New Flight Director 632Trending of Historical Data 632Providing Limited Availability 633High-Availability Sites 634Summary 636Exercises 637Chapter 35: Centralization Overview 639Rationale for Reorganizing 640Approaches and Hybrids 642Summary 643Exercises 644Chapter 36: Centralization Recommendations 645Architecture 645Security 645Infrastructure 648Support 654Purchasing 655Lab Environments 656 Summary 656Exercises 657Chapter 37: Centralizing a Service 659Understand the Current Solution 660Make a Detailed Plan 661Get Management Support 662Fix the Problems 662Provide an Excellent Service 663Start Slowly 663Look for Low-Hanging Fruit 664When to Decentralize 665Managing Decentralized Services 666Summary 667Exercises 668Part VIII: Service Recommendations 669Chapter 38: Service Monitoring 671Types of Monitoring 672Building a Monitoring System 673Historical Monitoring 67438.4Real-Time Monitoring 676Scaling 684Centralization and Accessibility 685Pervasive Monitoring 686End-to-End Tests 687Application Response Time Monitoring 688Compliance Monitoring 689Meta-monitoring 690Summary 690Exercises 691Chapter 39: Namespaces 693What Is a Namespace? 693Basic Rules of Namespaces 694Defining Names 694Merging Namespaces 698Life-Cycle Management 699Reuse 700Usage 701Federated Identity 708Summary 709Exercises 710Chapter 40: Nameservices 711Nameservice Data 711Reliability 714Access Policy 721Change Policies 723Change Procedures 724Centralized Management 726Summary 728Exercises 728Chapter 41: Email Service 729Privacy Policy 730Namespaces 730Reliability 731Simplicity 733Spam and Virus Blocking 735Generality 736Automation 737Monitoring 738Redundancy 738Scaling 739Security Issues 742Encryption 743Email Retention Policy 743Communication 744High-Volume List Processing 745Summary 746Exercises 747Chapter 42: Print Service 749Level of Centralization 750Print Architecture Policy 751Documentation 754Monitoring 755Environmental Issues 756Shredding 757Summary 758Exercises 758Chapter 43: Data Storage 759Terminology 760Managing Storage 765Storage as a Service 772Performance 780Evaluating New Storage Solutions 784Common Data Storage Problems 787Summary 789Exercises 790Chapter 44: Backup and Restore 793Getting Started 794Reasons for Restores 795Corporate Guidelines 799A Data-Recovery SLA and Policy 800The Backup Schedule 801Time and Capacity Planning 807Consumables Planning 809Restore-Process Issues 815Backup Automation 816Centralization 819Technology Changes 820Summary 821Exercises 822Chapter 45: Software Repositories 825Types of Repositories 826Benefits of Repositories 827Package Management Systems 829Anatomy of a Package 829Anatomy of a Repository 833Managing a Repository 837Repository Client 841Build Environment 843Repository Examples 845Summary 848Exercises 849Chapter 46: Web Services 851Simple Web Servers 852Multiple Web Servers on One Host 853Service Level Agreements 854Monitoring 855Scaling for Web Services 855Web Service Security 859Content Management 866Summary 868Exercises 869Part IX: Management Practices 871Chapter 47: Ethics 873Informed Consent 873Code of Ethics 875Customer Usage Guidelines 875Privileged-Access Code of Conduct 877Copyright Adherence 878Working with Law Enforcement 881Setting Expectations on Privacy and Monitoring 885Being Told to Do Something Illegal/Unethical 887Observing Illegal Activity 888Summary 889Exercises 889Chapter 48: Organizational Structures 891Sizing 892Funding Models 894Management Chain’s Influence 897Skill Selection 898Infrastructure Teams 900Customer Support 902Helpdesk 904Outsourcing 904Consultants and Contractors 906Sample Organizational Structures 907Summary 911Exercises 911Chapter 49: Perception and Visibility 913Perception 913Visibility 925Summary 933Exercises 934Chapter 50: Time Management 935Interruptions 935Follow-Through 937Basic To-Do List Management 938Setting Goals 939Handling Email Once 940Precompiling Decisions 942Finding Free Time 943Dealing with Ineffective People 944Dealing with Slow Bureaucrats 944Summary 946Exercises 946Chapter 51: Communication and Negotiation 949Communication 949I Statements 950Active Listening 950Negotiation 954Additional Negotiation Tips 958Further Reading 960Summary 961Exercises 961Chapter 52: Being a Happy SA 963Happiness 963Accepting Criticism 965Your Support Structure 965Balancing Work and Personal Life 966Professional Development 967Staying Technical 968Loving Your Job 969Motivation 970Managing Your Manager 972Self-Help Books 976Summary 976Exercises 977Chapter 53: Hiring System Administrators 979Job Description 980Skill Level 982Recruiting 983Timing 985Team Considerations 987The Interview Team 990Interview Process 991Technical Interviewing 994Nontechnical Interviewing 998Selling the Position 1000Employee Retention 1000Getting Noticed 1001Summary 1002Exercises 1003Chapter 54: Firing System Administrators 1005Cooperate with Corporate HR 1006The Exit Checklist 1007Removing Access 1007Logistics 1011Examples 1011Supporting Infrastructure 1014Summary 1015Exercises 1016Part X: Being More Awesome 1017Chapter 55: Operational Excellence 1019What Does Operational Excellence Look Like? 1019How to Measure Greatness 1020Assessment Methodology 1021Service Assessments 1025Organizational Assessments 1029Levels of Improvement 1030Getting Started 1031Summary 1032Exercises 1033Chapter 56: Operational Assessments 1035Regular Tasks (RT) 1036Emergency Response (ER) 1039Monitoring and Metrics (MM) 1041Capacity Planning (CP) 1043Change Management (CM) 1045New Product Introduction and Removal (NPI/NPR) 1047Service Deployment and Decommissioning (SDD) 1049Performance and Efficiency (PE) 1051Service Delivery: The Build Phase 1054Service Delivery: The Deployment Phase 1056Toil Reduction 1058Disaster Preparedness 1060Epilogue 1063Part XI: Appendices 1065Appendix A: What to Do When . . . 1067Appendix B: The Many Roles of a System Administrator 1089B.1 Common Positive Roles 1090B.2 Negative Roles 1107B.3 Team Roles 1109B.4 Summary 1112Exercises 1112Bibliography 1115Index 1121Authors BiographyThomas A. Limoncelli is an internationally recognized author, speaker, and system administrator with more than twenty years of experience at companies like Google, Bell Labs, and StackOverflow.com.Christina J. Hogan has more than twenty years of experience in system administration and network engineering, from Silicon Valley to Italy and Switzerland. She has a master’s degree in computer science, a doctorate in aeronautical engineering, and has been part of a Formula 1 racing team.Strata R. Chalup has more than twenty-five years of experience in Silicon Valley, focusing on IT strategy, best-practices, and scalable infrastructures at firms that include Apple, Sun, Cisco, McAfee, and Palm.