A while back I was developing a workflow scheduler. However the run duration kept coming back negative. The runs kept finishing like 15 seconds before they started. Eventually I figured out why; the company had configured a policy to use their custom NTP server, but it was 16.7 seconds off. So my Windows laptop’s time was in the future, but the WSL VM (which ran the program) used another NTP server and was correct. In the end I just added a bodge that made the program request the time from an external server on startup to calculate an offset.

  • IphtashuFitz@lemmy.world
    link
    fedilink
    English
    arrow-up
    2
    ·
    46 minutes ago

    Back in the 90’s I had to debug a printer routine in a reporting tool. It would occasionally crash. Only on Wednesdays. Only on Wednesdays in September. Only on Wednesdays in September after September 9th.

    Code was written in c.

  • expletivedeleted@lemmy.zip
    link
    fedilink
    arrow-up
    4
    ·
    2 hours ago

    I was hired to work on a CMS designed for schools. I was tasked with debugging.

    I started off with the newest issues and worked my way back.

    Well, there was a bug where teachers would upload a video but it wouldn’t play on the page. It was plaguing a team of 8 for YEARS.

    It was mentioned in our morning standup, and I decided to fast track it.

    The dude who wrote the code was shrinking the display size in half: well how do you think that would work when a dimension of the video isn’t divisible by 2?

    So, remember, I am the new guy, quickly fixed it using an if/then and the % modulus operator, and lo and behold, the next morning standup I had to explain the fix in front of the project manager, his boss, and the entire team…

    fucking awkward. “You see, there’s this thing in division called a remainder.” This was the first thing in the morning.

    But because the bug had been so ever present, that’s why the boss’s boss attended the standup. God, I just wanted to die.

    • cristian64@reddthat.com
      link
      fedilink
      arrow-up
      2
      ·
      1 hour ago

      I don’t think I understand what the issue was or what the fix was. Was it a language or system that cannot perform the division with truncation?

      • bss03@infosec.pub
        link
        fedilink
        English
        arrow-up
        1
        ·
        27 minutes ago

        I think there are some versions of Python and/or PHP where the normal division operator silently promotes to IEEE floats, and you have to explicitly cast back to truncate/floor the value.

  • ExtraMedicated@lemmy.world
    link
    fedilink
    English
    arrow-up
    10
    ·
    3 hours ago

    Internet Explorer didn’t automatically have “console.log” defined until you opened the console window. So there was an error breaking the web page that would disappear when you tried to open the console to see the error. The error was that console.log wasn’t defined. Fucking IE.

    • bss03@infosec.pub
      link
      fedilink
      English
      arrow-up
      3
      ·
      2 hours ago

      “Yoda conditionals” are a must in languages where bindings/references are mutable by default. E.g.: “if (0 == size)” or “if (0x4269 == magic)”. If you write ‘=’ instead of ‘==’ you get a compiler error, so always put the constant first.

    • naur@tech.pr0n.pl
      link
      fedilink
      English
      arrow-up
      5
      ·
      4 hours ago

      I once spent several hours debugging a performance issue in the Linux builds of a project. The service worked fine on Windows but couldn’t keep up with the traffic when deployed on Linux. Interestingly, CPU load was actually higher on Windows, and no sampling profiler showed any suspicious hot spots.

      It turned out the problem was in the program’s main loop, which was OS-specific and called something like sleep(float seconds) (yes, I know…). The parameter was set to 1e-3 on Windows and 10e-3 on Linux. I stared at that line several times before I noticed the difference.


      Another time, all services of a distributed system were regularly reporting a loss of communication at 4 AM. The servers were supposed to receive data continuously, so even a few seconds of silence was enough to raise an alert.

      Half of the team investigated all kinds of possible locking scenarios (the applications were multi-threaded). Then we noticed that all the logs in the system (including the kernel’s) had the same gap in timestamps.

      Then someone remembered, that the customer had a cron job which did some maintenance at night. They confirmed that it halted the entire VM with all services for about 5 seconds.

  • yaroto98@lemmy.world
    link
    fedilink
    arrow-up
    10
    ·
    5 hours ago

    Was told to test a new database migration as they updated from mainframe to a current version of Oracle. First I was to test data integrity. Wrote a python script to pull from the old db and verify that record matched the new. Found sooooo many bugs, sent it to the dev. Half were real bugs, half were bugs with my script. Turns out they didn’t actually want a 1:1 on the new db they were going to change things cuz they could. Lots of back and forth and by the end my python script essentially mirrored the production code.

    Then they wanted me to load test the new system. I wrote a quick script to query N rows from the db with X threads and blast the new service. Didn’t take long to write, presented my finished script and plan to pull in manual testers to also run my script as needed.

    Dev was cocky because they’d setup auto scaling on AWS. Manager was concerned I’d taken time to write a script instead of just spamming the same message over and over with a tool like gatling.

    First run I took down the system very quickly, like within a minute with just my laptop. Note, this was a system that was planned to handle 100s of millions of requests an hour. Dev was impressed, turned out to be a auto scaling misconfiguration. Second time it took a few mins but it died again. Config needed some more tweaks. Third time I took it down because of some queueing issue that raised it’s ugly head. Fourth time I took it down they found a bug in the code. Fifth, Sixth, and Seventh time I needed the manual tester’s help, but the devs were scratching their heads trying to figure out how I was taking them out in 5mins of sustained load. Took two teams of devs scratching their heads and several itterations through bringing the system down and even two separate devs walked through my code to see what on earth I was doing. To top it off it wouldn’t just take down the microservice, it would crash our entire AWS infrastructure for the company. Not just our project but all our aws projects in the building went down.

    They finally found the problem. The silly little problem. Remember that data I pulled from the db and blasted them with? Well, we hit a poision pill situation. One of the records turns out was a manual tester’s favorite to use when running one of his tests. That one record had 30,000 attached sub-records. Everytime that tester ran that test case he just kept adding a sub-record to it. Dude didn’t know he was slowly crafting such a massive data bomb all those years that would take down all our AWS servers.

    They had also fixed the problem with one service crashing the others. And that my friends is why you never skimp on good automated testing. When it went live there was no major issues or outages, though we did let it soak in parallel with prod for weeks to make sure we trusted the results.

  • HamsterRage@lemmy.ca
    link
    fedilink
    arrow-up
    7
    ·
    6 hours ago

    Whenever I book a flight with Air France, I have to enter our birth dates off by one day. We’re in Canada, and the Air France servers are most likely in…France, or at least using a french time zone.

    Anyways, I figure that someone decided to use some form of LocalTime midnight for the date. We’re 6 or 7 hours later, so when it translates it to French time, it’s a different day.

    So, not a bug that I fixed, but one that I work around. It’s been this way for years, too.

  • mkwt@lemmy.world
    link
    fedilink
    arrow-up
    14
    ·
    7 hours ago

    This was on a Fortran project that had no encapsulation and way too many global variables.

    Suddenly, one particular calculation was failing, and not in a floating point rounding error kind of way. After setting up several scenarios to generate various input values, I determined that Fortran’s absolute value function was returning the wrong results.

    I couldn’t understand why, so I went to the Lead Architect. He poked at it for about four minutes and did some searches, and found where someone else had defined a global variable named ABS that was masking the built-in abs function. The function call argument [e.g. abs(3)] was being compiled as an array index on this ABS variable [which was not an array, so yay, undefined behaviour].

  • AstroLightz@lemmy.world
    link
    fedilink
    arrow-up
    3
    ·
    5 hours ago

    I had a project I was working on a while back. I kept testing it and trying to find the bug that was causing my output to be completely wrong. It took me about 2 days to finally find it:

    I had a boolean set to false, when it should have been set to true.

    That one little bug completely broke my sorting algorithm. After changing it, it worked fine. :/

  • carrylex@lemmy.world
    link
    fedilink
    arrow-up
    10
    ·
    7 hours ago

    IT department really wanted to use their snake oil antivirus enterprise super secure hackproof™ software everywhere.

    So it was installed on the CI servers.

    Fastforward a few months and we were wondering why one build agent was no longer picking up jobs.

    After a day of debugging we found out that a certain snake oil sofware tampered with a single machine internal TCP conncetion and somehow fucked it up in a way that all packets in one direction just vanished.

    Disabling the snake oil software magically fixed the problem.

    • HamsterRage@lemmy.ca
      link
      fedilink
      arrow-up
      3
      ·
      6 hours ago

      Should a CI server even have access to the Internet and need antivirus software? Maven access I guess.

      • bss03@infosec.pub
        link
        fedilink
        English
        arrow-up
        2
        ·
        2 hours ago

        It is good to have a build method that doesn’t require network access; at one time that used to be required for Debian builds. However, for a variety of reasons, most CI servers do have access to the network, even if there’s a firewall (or NAT) between them and the Internet.

  • Bananskal@nord.pub
    link
    fedilink
    English
    arrow-up
    7
    ·
    6 hours ago

    Took a course during uni on computer graphics and one assignment I handed in needed correction, so I spent the better part of a summer trying to debug and find the flaw. Difficult to debug the graphics pipeline back then and with basically no tutors on hand during vacation times.

    Turns there was nothing wrong with my thinking and program logic, I had just accidentally written two adjacent lines in the wrong order.

    I transposed them, everything worked.

    I was both relieved and pissed off at the same time. So clumsy.

  • bacon_pdp@lemmy.world
    link
    fedilink
    arrow-up
    3
    ·
    5 hours ago

    They did a 400 table join to return ‘true’ to test to see if they connected to the database; it took 48 minutes to run. Printed out, it took up 38 sheets of paper.

    I replaced it with a dozen sequence checks and individual select statements for each table; it took 3 seconds to run. And it could accurately identify which tables and columns are missing. Printed out, it fit on a single page.

  • lime!@feddit.nu
    link
    fedilink
    arrow-up
    66
    ·
    11 hours ago

    i was just a user, but i debugged a weird issue where one of those blue light reduction programs would start flickering and inverting day and night time during june.

    turns out the dev had made the assumption that the sun always rises before it sets, and i was far enough north that the sun would start setting after midnight.

  • sunbeam60@feddit.uk
    link
    fedilink
    arrow-up
    15
    ·
    10 hours ago

    Two variables named somethingl and somethingI that had three grown developers scratching their head why the wrong variable was getting updated. This was back in the day when intellisense and colouring was way less advanced.

    An actual compiler bug that wrote into the wrong register, went all the way down to assembler to realise the issue. This was on the early Xbox 360 (PowerPC) compiler.

  • Fargeol@lemmy.world
    link
    fedilink
    arrow-up
    29
    ·
    11 hours ago

    I found an issue in a software used by government employees.
    A request would fail if the server and the client had different years in their system time which could happen if they were located in different time zones.
    After a few calculations I asked: “Hey, what are the chances a government employee would do this important task between 6pm and midnight on December 31st?”
    We decided not to prioritize the issue.

      • qaz@lemmy.worldOP
        link
        fedilink
        English
        arrow-up
        4
        ·
        6 hours ago

        “Hey, what are the chances a government employee would do this important task between 6pm and midnight on December 31st?”

        I guess “this important task” could refer to firing a nuke, which to be fair is unlikely to happen at that moment in time.