Inspired by the very similar thread about school incidents.
INC-224, never forget.
I am an infra engineer at a fairly large scale (not like Amazon, but we have some BIG customers) SaaS company; despite our scale, we are only like 250 people and of them only about 90 engineers. We store a bunch of data in MySQL.
15:30:00, I get a page “MySQL table is full.” I immediately know my day is ruined, since I’ve never heard of this error before, but know it ain’t great.
15:30:10, every Pagerduty escalation policy in the entire company gets bombarded with pages.
I look at the database instance. The table size is “only” 16TiB, so it’s a bit confusing.
We are hard down for several hours as we scramble to delete data or somehow free up space. Turns out, google backs ClpudSQL MySQL instances with ext4 disks instead of zfs, and the max file size on ext4 is… you guessed it, 16TiB.
We learned a LOT of lessons from this, and are now offloading a shitload of json into either MongoDB or gcs, depending on the requirements. The largest table is down to 3TiB now :D
I love it.
All the other comments had guns sex and drugs.
Your story had mySQL.
Mad lad.
I understood almost none of that.
Database (thing that holds and retrieves bunch of data) broke when it reached a size of 16 Terabytes because the underlying filesystem (Thing that lets you store data on a physical disk like a hard drive or SSD) has a maximum possible size of 16 Terabytes by default (ext4)
16 TiB is roughly 16,000 Gigabytes which is roughly 16,000,000 Megabytes
Ty. I understood the tb but I didn’t know what a lot of the other abbreviations meant.
An IT contractor at my government job was one of the people that tried to kidnap Michigan governor Gretchen Whitmer.
Previous HR was well beyond retirement age essentially working to have something to do and one day emailed all of management a spreadsheet asking us to verify our information. That sheet contained each of our full names, addresses, phone numbers, birth dates, social security number, etc.
To my knowledge nothing of significance happened. I have my credit frozen.
I worked for a company that handled a ton of personal data. Pretty much every person in Germany, including addresses, bank account details, etc.
On my first day there (fresh from university) I was given literally full read access to the entire database. And as I later found out by accident: they did not track any data exfiltration at all. I copied several gigabytes of data without anyone noticing.
Your data is only as secure as the least motivated data broker sees fit. And that’s not very fit.
A few years ago I asked a customer for a list of employees, so I could verify who could purchase on their account. They replied with their personnel files. Luckily it didn’t have social security numbers, but it had a LOT of personal information. Medical records, drug test results, stuff like that.
The whole workplace drug testing thing is so wild to me. An employer can actually lay claim to your bodily fluids? Absolutely mental.
In the Netherlands, it’s very simple:
- if there are performance problems, then you address your employee’s performance problems.
- if there are no performance problems, then there is no problem and what your employee does in their free time is none of your business.
Had an executive assistant at my company who did very little if anything. Nobody knew why she was kept around and paid so much. Everyone pressured the CEO to fire her, but he strongly resisted. Eventually she was fired, but immediately threatened to sue for sexual harassment. CEO threw her a lovely settlement check despite claiming that nothing ever happened. Mmhmm.
Software company before git. The source server corrupted and the product code was lost. 5 guys had to get together and figure out the latest version between them (everybody had different changesets) and produce a new “current” version. At the end we lost all history prior and ever since all changes prior to 2008 have been attributed to 1 guy.
I used to work at an accounting/consulting firm who were dead set on writing business applications in VBA within Excel. The code was embedded in the notebook, and to distribute the software was sending the latest version of the Excel file. This made version control virtually impossible, and we would instead combine our work manually.
I cannot recommend having tech-illiterate people lead software projects.
The amount of times I hear people telling me that “I should just do it in Excel”. Excel. Is not. A database.
Excel is a single-assignment dynamically-typed functional programming language with a really obtuse editor.
Stop… Stop… I’m already dead
Close enough when your actual database system is written in fucking COBOL.
Excel is a whole OS unto itself. Like Emacs except you can get out of it.
Subversion has existed probably for longer than your company, the fucking managers couldn’t be arsed to read a damn book?
They were using SourceSafe back then. But any source control that isnt decentralised has the same problem. If the central server gets deleted so does all history
I had a worse experience. My first internship was doing web development in ColdFusion. Why that language? Because when the company was first starting, none of the funders wanted to learn Linux/Apache administration and CF ran on Windows.
Also, the front end development team did not have version control but shared code via a file server.
More impressive than the fact that you saved a repo once is that the same repo still exists today with the complete git history. At the rate companies abandon products for new ones, old repos are rare.
Our repo is old as time. Carried through from SourceSafe to TFS to Git
Gotta respect that save. Reminds me of the Toy Story 2 assets being lost from a server failure and they were saved by one employee having a copy on their personal computer at home.
It wasn’t a server failure. Someone rm -rf on the root of the server. The server did what it was told.
I hurt my back working today. That was an incident.
That time the food cart pod exploded:
https://www.oregonlive.com/portland/2017/10/food_cart_pod_on_fire_in_downt.html
Them including that pic of Audrey Donovan is hilarious
Our department sometimes had a few interns, most of them young and female. Usually one of them got her workplace in the boss’s room in the office and he had plenty of time to show them how things are done etc.
One day the boss invited all staff to his house for a nice little summer barbecue. Later in the evening we recognized him being absent from the party for nearly 2 hours, and one of the interns was missing for exactly the same time.
Worked at a place where our CIO was completely unqualified to be a leader, much less a leader in IT. She was a micromanager who took the position of “telling stakeholders” instead of “working with stakeholders” so any project she was on was really her pushing through whatever agenda she had at the time. Meanwhile her deputy CIO was stealing computer equipment from the server room but I digress…
April fools one year and I decide to prank it up. I moved the hinges (not the door handles) of the freezer/fridge in the breakroom so that the handle and hinges were on the same side. It’s a fifteen minute job to move everything so I did it the night before the 1st.
The next morning our hungover CIO stumbles into the breakroom and cannot get the fridge to open. After a few seconds of futile tugging on the handle, she gave up and took her lunch to her office.
Others in the office figured it out pretty quickly and had a good chuckle.
Later on that day CIO sends out a nastygram about pranks being unprofessional, property damage, someone was going to be in huge trouble, yadda yadda…
But she’s not the director. The director tells her to basically fuck off, it was a funny prank, and perhaps she needed to lighten up.
She never found out it was me.
Ha!! As an appliance repair guy i learned about reversing the door hinges+handles a long time ago. It never occurred to me to use it for a prank until i was living in my apartment for a few years, and realized it really would make more sense to reverse the hinges to open the door the other way. I moved the hinges, but then it occurred to me that i can leave the handles where they were and prank all my friends when they came over. Unsurprisingly, it works! People usually would figure it out eventually but sometimes we had to intervene if they were getting too rough with it.
I got so used to having it set up that way that once in a blue moon I’d go to open other people’s refrigerators the wrong way (not the best look for a repair tech, LOL)
The overnight IT guy was caught watching porn while working (this was over a decade ago, he was in the office every night and not a remote worker). How was he caught? He was saving the pornographic photos on a shared network drive…
When confronted, he didn’t try to deny anything, his explanation was simply, “That’s just my thing.”
One of my very first tech jobs, there was a guy who watched porn in his office, as far as I could tell just continuously. I saw the DNS logs so I was aware. I didn’t care as long as it wasn’t interfering with my ability to get things done.
One day I had to go in his office and talk to him about some of his code that was breaking, and he went OH and tried to hide all the blatant porn on his screen. Like dude I know. I don’t care. I am here to talk with you about your shitty code not your personal failings and issues; those are purely your own problem IMO.
The ceo of a client viewed porn pretty much all day every day during the pan when everyone was wfh. Our screen sharing tool showed a stamp-sized preview of the client device before connecting. One of the interns said after over a week of trying to engage him with the chat feature and closing the connection due to porn before sending a chat, he gave up and accepted that there was no time when porn wasn’t on-screen.
I saw this so often when I was client facing. CEOs, doctors, and sales people were the biggest offenders.
We had a gyno who had a huge pile of porn on his file server. It was all from the waist up. Seriously, he had half a terabyte of titty pics.
Separately, there was a sales guy who was juggling like 5 women (poorly) at any given time. He was fucking gross and would try to show them off to anyone who came to work on his continuous computer problems that were all caused by him.
Separately from that, we had a “troubled boys ranch” as one of our clients. One of the C Suite was caught with porn and we had to go over it with a fine toothed comb to make sure none of it was of any of the kids. There wasn’t (thankfully) but there was a whole lawsuit about it and he was charged with showing it to some of the kids.
A gyno with just tit pics either feels like he went 50/50 in career path and made a bad choice, or has become so desensitised (or put off) by his work that he just cant bear to look below the belt
When your hobby becomes your job!
Boss: “Were you looking at porn in the office?”
IT guy:
We should be able to look at a liiittle porn at work.
You’re a rock star!
Aside from the obvious don’t watch prom at work, I don’t see how anyone does it with company property. Doesn’t pretty much everyone have a phone or computer these days? Shouldn’t it be common sense to use your phone rather than the work computers need by your employer?
As a treat.
I would say no. But there is a part of me that sees he’s an overnight IT worker, and then I’m like “what the hell else would he be doing?”
WoW
EvE Online until he perfecta his spreadsheets
As a former overnight IT worker, I always just assumed my browsing history would be reviewed by other IT people. Maybe they wondered why I spent so much time on horror game forums, maybe they already knew I was a disturbed mind.
My company called all lab staff “pandemic heroes” for coming in every day during the pandemic and taking on extra work to compensate for management and office staff who stayed home for years.
Then shortly after return to office, they closed the lab and laid off all lab staff.
Sounds like your company took the Veterans Affairs approach to “hero response”.
Worst part is that they did it mostly to boost the IPI right before we went public by driving down operating costs.
We weren’t even able to buy in u til 6 months after going public and the price leveled off at 6 months
Guy found a gun in the customer’s stuff
Guy starting waving it around and playing with it, pulled the fuckin trigger, almost shot one of his coworkers
Cops came, guy said he was moving a cabinet and it went off which obviously no one believed, somehow he wasn’t arrested, idk
Guy was fired over the phone before he left the customer’s house
Was this one on the news? This is very very familiar
Another:
Big awful dude starts working, among other issues he was SUPER upset that the girls at the gym are allowed to have their own separate area to work out where he can’t ogle them, he felt this was grossly unfair and was angry about it
So anyway my boss goes back to the truck to get something, at like 9 in the morning on the job site, opens up the back, the ENTIRE truck is filled with weed smoke which billows out because big awful dude is in there getting high. Boss is upset, obviously, but big awful dude is just laughing
I think they had to finish out the day with him but the boss was definitely irritated about it
Oh shit! I forgot one from another job.
One of the busboys walked into the office, found no people and a satchel with about $30,000 in cash, picked it up and walked out, clocked out like normal, went home.
Guy SHOWED UP TO WORK THE NEXT DAY. Just assuming I guess, they won’t have cameras or anything, if I just don’t say anything there’s no way they can know who it was and they’ll probably just move on if I play it cool.
I guess the management was pretty aware of his level of planning skills because they had cops waiting at the restaurant at the time of his scheduled starting time and he was taken away in cuffs, presumably not to return for quite a long time.
I guess in his defense, he knew damn well if he stopped coming to work the day after $30k went missing, they’d know it was him.
I mean obviously the smart thing to do is not to fucking touch the money, but I’ll give the guy showing up to work the next day. It’s not like $30k is flee-to-Argentina-and-start-a-new-life money.
No, the smart thing to do is to not leave 30k unattended around people who aren’t paid well.
I was supervising filling in a pit we had dug on the edge of a forest. We had dump trucks coming in dumping gravel. One particular driver wasn’t great at his job and there had been issues with him in the past.
That driver came in and dumped his gravel, but then he drove off with his bed still raised and almost immediately smashed into electric lines that ran off into the forest. One telephone pole even snapped at the base and fell over.
Within 30 seconds multiple cops came speeding onto the job site. It turns out those electric lines ran to a radio tower in the woods that ran the police radio. The idiot in the dump truck had taken out the police comms for the whole town.
Note: if you’re planning a crime in that town, you only have to cut one wire to disable all police communication.
That’s some lacking infrastructure
You’d be surprised, how fragile critical infrastructure often is. There was an incident in Europe a few years ago, where a single miscalculation in a planned power line shutdown almost caused the entire European grid to split.
It slowed down a bit, and then we quickly learned that maintaining the perfect 50hz wasn’t actually necessary anymore. Few people still have clocks that depend on it
I’m not talking about the incident in Romania, but in Germany.
A shipyard needed some wires over a river deactivated and that caused an overload cascade, because the river was the border between two providers who had different assumptions about the capacity of the power lines connecting them.
oh damn, ain’t something. I will be looking into that, thank you!
Clocks, true.
Computer systems in general, however, will start acting very squirrelly outside of an approved MHz range. Wall warts and power supplies can handle only so much deviation from the norm. It’s why high-end UPS systems do power conditioning to provide a pure sine wave.
No worries we’re talking like 49.9995 hz
That’s some lacking infrastructure
They probably had plenty of infrastructure for normal operations.
What they were lacking was a BCDR plan.
…which includes having backup lines or a more robust installation. Police officers aren’t engineers or system administrators for public infrastructure.
You’re right tho, a backup alone would not be sufficient
This is why we shld bury our lines, much more effort to dig down six feet than get a ladder and snip
And this is how a micro quake severed our T1 line from LA to Phoenix and shut the network down in our office for a week.
Honestly never thought of that, sounds like there would need to be some sort of protective channeling, with space to allow some shifting
Buried lines of all kinds are frequently severed by excavators because their position isn’t properly or fully documented.
The best set up I ever saw was a sewer tunnel, almost 12 feet tall, that handled all the services. From sewage to water to electricity to data; it held everything and was trivial to maintain and run new lines in.
line sounds like a really interesting idea, although I feel like documenting where you put things should be a basic task. Probably why it’s not done properly
A wild backhoe appears!
What does a network engineer bring on a hiking trip in the woods? Water, snacks, extra sunscreen, a first aid kit, bug repellent, bear spray … and a folding shovel and a piece of fiber-optic cable.
(What’s the fiber for?)
Well, if you get lost in the woods or need to be rescued, you take the shovel, dig a trench, put the fiber in it, bury it … and within an hour, someone with a backhoe will show up to tear it up. Then you can just follow the backhoe tracks back to civilization.
It used Dig! It’s Super Effective!
Couldn’t they sell a few of their spare MRAPs to buy a backup generator and a redundant microwave link? Sheesh.
One day a coworker of mine was walking into our huge office building and thought he saw a mitten on the ground of the lobby. When he picked it up it was actually a pair of lacy women’s underwear. Ostensibly it fell out of someone’s gym bag or got caught in their pant leg in the laundry and dislodged there. He drops it immediately and comes into the office. He doesn’t mention this to anyone.
Two hours later the main receptionist comes in with the underwear in front of our whole group and says she saw him drop these this morning and she wants to return them. He’s denying the whole thing and at this point none of us have the previous context and all locked in to the conversation and silent laughing. She says, “We just want to give these back in case they have sentimental value!” and the the whole group is dying laughing now. He eventually convinces her he isn’t interested in a stranger’s underwear (which she bare handing) to which she says she’ll keep them in case he changes his mind (???).
It’s been 5 years and it gets brought up nearly daily
Does he still work there and does he laugh with you? Otherwise this sounds like bullying
Sounds like the receptionist did this on purpose lol
Someone must’ve summoned Shenron the day before he found them and got underwhelmed by the wish fulfillment.
Oolong back on his shit
Funny if they were her panties all along. Turned the embarrassment from “Guess who dropped her panties in the lobby” to " Guess who was playing with panties in the lobby."
This adds a whole new dimension to the lore.
Depends. Had a client pull a knife on me once, and another dragged me around the facility for an hour while he tried to break down a door to “kill” another client because he had stolen the change from a $5 Taco Bell gift card.
The other incident being was a coworker harboring one of the fugitive kids at her house with her like…6 children while her husband was away in Nebraska for work. Randomly saw her in family court a year later while I was working another job, hopefully while her husband fights her for custody of the kids…
