Mr_KrzYch00's Analysis

Stable Diffusion 3 and the possible why

2024-06-18T04:52:00.016+02:00

I will begin by citing:

"5.3.1. DATA PREPROCESSING
Pre-Training Mitigations Training data significantly impacts a generative model’s abilities. Consequently, data filtering is effective at constraining undesirable capabilities (Nichol, 2022). Before training at sale, we filter our data for the following categories: (i) Sexual content: We use NSFW-detection models to filter for explicit content. (ii) Aesthetics: We remove images for which our rating systems predict a low score. (iii) Regurgitation: We use a cluster-based deduplication method to remove perceptual and semantic duplicates from the training data; see Ap pendix E.2."

Source: https://arxiv.org/pdf/2403.03206

While I do not personally agree or disagree on the matter, in science the most important thing would be for researches to be conducted free from any external pressure that may impact such research's outcome. Otherwise we may be crossing a border between science and other areas, whether it being a concern, social matter or any other matter causing science retardation.

My questions would be:

- What the responsibility of parties involved in pressuring researchers is?

- Is there any form of cooperation between researches and pressuring entities?

- Is there any responsibility of pressuring parties towards the failed research experiment due to such pressure being conducted (be it financial, expertise or any form that guarantees the success of given research)?

While the release of Stable Diffusion 3 couldn't entirely be called a "failed experiment" per se, the AI struggles to properly represent human subjects while having much less or none issues in other areas of image generation. Trying to make the AI safe most likely caused a huge problem with the said AI being underfit in the proper representation of human subject, rendering it finally unsafe for the viewer in certain circumstances. From accidental "nakedness" we shifted towards accidental horrible mutations when trying to represent a human subject that is not in the casual standing position. This could be well seen as a regression.

If a research is conducted by a company that was put under any form of pressure and the said company is known to produce research papers, what is the protection for such company against any form of issue that the pressuring party has therefor caused? As well as, what is the liability of the pressuring entity if the pressure such entity conducted caused problems of any nature to the research company in question?

My own research in the area has proven to me that AI such as Stable Diffusion can perform zero-shot tasks in answering various questions. When SD 1.5 model is asked how the hand should look like, it usually creates a hand containing 8 fingers. The result changes basing on the question related to the "hand" is asked (a "single" hand can have 4 fingers), usually never rendering a hand with 5 fingers. However, when asked specifically what a "hand's silhouette" should look like, the AI locked to the same seed responded by generating the most proper hand a human subject should be having, the hand with five fingers and of a proper hand structure, a black shadow-like hand however - various questions regarding a hand with "silhouette" word included was confirming the outcome for me.

Furthermore, if one very closely watches on how Stable Diffusion generates a hand, there is a clear sign of AI confusing reflections and shadows as additional finger structures. Zooming in closely in every image sample will reveal that, even if the hand seemingly has 4 or 5 fingers, there may be opaque fingers still visible near it, or whole hand structure generating itself under the initial hand structure. My conclusion was that the AI struggles to differentiate a normal hand from:

- clasped hands,

- hand shadows (imagine a hand's finger shadows on a cloth), and,

- hand reflections (imagine a well polished desk that reflects parts of the hand back).

Using the same methodology I conducted a small research with SD 3.0. The initial question I asked using T5XXL text encoder was "how a laying down on the floor subject should look like?" The AI responded by generating a dog subject which may give some sort of insight that the AI is overfit with animal subject (specifically dogs) and underfit with human subjects in this area of its knowledge - a shift of data distribution in the training set most likely caused by removal of data that is initially guaranteed or guarantees the success of a research (there might've been unexpected false-positives in detection as well, reducing a valid data that might've helped AI better learn what it is meant to learn).

Furthermore, when asked the same question but specifying that the subject is a "dog" and a "person", the AI answered in the following way:

(samples generated from ComfyUI, using the recommended selection of nodes with separate SD3 clip models, using FP16 version of T5XXL - seed locked)

While a dog subject is rendered without much of a trouble, it is very much noticeable that the result on the right is hardly a body of a human. Interestingly, asking for "human" renders very close representation of the pose the human subject supposedly has when laying down on the floor. By looking at the dimensions of the subject represented by the example on the right side, we can clearly assume that AI tries to "fit" the dimensions of a human body into the dimensions of a typical dog laying down on the floor in whatever means necessary. If the body of a human has to be completely deformed and causing absolutely horrific results, SD3 will gladly do so, maybe in the name of being "safe" or maybe something else that went wrong.

Note that we didn't provide any very specific prompt here, just those two simple questions. While some people advice to be more descriptive, if it was actually a must for SD3 to always have a descriptive input the AI wouldn't render the dog properly either, or render a STOP sign properly by being asked just that. One could argue that a dog laying on the floor is very common, but that's not how one should look at the dataset. We are not putting there anything we can find and call it a day, it has to be properly balanced and reproduction of the said dataset confirmed. Any anomalies in verification samples need to be noted down and observed over the course of training the AI. If the situation doesn't improve or, in worst cases, degrades, the dataset would require an urgent verification - it can as well turn out that it got accidentally corrupted and it's not yet too late to fix.

The need to be descriptive in regards to human subjects means that the AI cannot self-guide itself towards some correct generic/default result in regards to humans, or is heavily confused/polluted/poisoned with the other data that is causing a serious bias towards rendering poses unacceptable for human bodies to find themselves in, to the point of generating extreme deformities. We could as well theorize that there were none human subjects laying on anything in the dataset after its pruning, so the AI has to guess how such subject could look like, being confident enough to try doing that and confident enough that human bodies can undergo a "very uncomfortable shapeshifting."

Obviously picking the correct resolution, and in turn an aspect ratio - as some suggest, does pay a major role in SD as observed with older models (e.g. SD1.5). It's good that the AI doesn't feel too confident to be shrinking or stretching bodies too much in most cases. However, it should be more confident to be cropping the image rendered and still render the body partially rather than render images in too unrealistic "morphed subject" situations that are hard to view, while the AI (probably) seeing it as a form of creativity and doing it when not explicitly asked for.

While the AI needs creativity to refrain itself from being too repetitive and boring, it cannot be passing the borders of sanity too often. Presenting a strange "insane" example every now and then is not in itself bad - see it as a serious person occasionally telling a joke. However, once the prompting is incapable of correcting such outcome at all or not in too tedious way, then it may be a sign that something went horribly wrong in how the AI trained itself and its knowledge is lacking or incorrect.

We could still argue that the anomalies are caused by inference being in its immature state and still needs improvements. Said so, I would be very surprised if such results are being caused by a bug of sorts. Though I wouldn't rule out that there could be something like certain weights being not processed or incorrectly processed, those possibly responsible for certain details. That would only be a theory though as I have no proofs of such being actually the case.

Finally, I myself was guessing that maybe the major problem is that the 2B model cannot handle the things the same way 8B would and possibly still required some babysitting to overcome its limitations. It could theoretically have not enough parameters to be paying correct attention to rarer examples in the provided dataset, filling itself up with what it could and being incapable of changing its ways of understanding later on to, for example, reconstruct proper human poses, limbs amount, so on so fort. To me it's clear that less number of parameters may reduce certain details, but it's unclear what those details may actually be.

While I'm not claiming if and in what part the issues with SD3 Medium are related to the below, my overall personal opinion on general purpose AIs, those that are meant to replicate non-specific task oriented results, is that they must be always trained on an unattended (but balanced or weighted) dataset. In the worst case scenario it could be a subtly modified dataset that is proven to not have any significant impact on the final outcome. However, the first method should be the most valid one.

I believe that artificial intelligence was proven to be very much comparable to any other form of intelligence - at least to the point we're trying to replicate human-like intelligence in artificial means. Human intelligence bases on associating one information with another to produce the desired thought outcome that can be then passed on or/and kept to oneself. If such intelligence lacks some form of critical information, the produced thought may be therefor seen incoherent to other recipients of the said thought. By recipient we do not necessarily mean another human but anything that is the recipient of such thought (a person, an animal, a situation, an inanimate object etc., a cause-and-effect action in general.)

Humans are further taught to behave in a particular way while still maintaining their knowledge. Similar methodics should be utilized in the field of AI. This could be post-filtering, fine-tuning or any sort of way that doesn't change the initial understanding of given AI on the matter but still will ensure such AI "refrains" from passing its "artificial thought" in concerning ways.

I urge people to calmly debate on things, know when to be sorry and work on finding the right solution. Rather than putting pressure on each other and enforcing one's way onto another's. It's so easy to scream at someone, it's so hard to calmly walk the path of mutual understanding.

With that I will end this post as there is nothing more to say. Please, by all means, do not see that post as an opinion that is meant to be forced upon anyone. I merely share my concerns and understanding on the matter.

Vampire Survivors - almost no dps, almost invincible

2024-01-31T23:48:00.017+01:00

Our special friends have trouble hurting the player ;)

When using 3 max weapons + XX arcana + Celestial Dusting + shield + freezing, in some cases there may be long periods of involuntary "invincibility". When the player wishes to die for revival bonuses, it may actually be hard to accomplish also for low DPS/AoE scenarios.

It also seems that the game is leaking memory when Celestial Dusting gets too powerful. Possibly flowers that get off screen are not properly freed from memory, causing the memory usage to keep growing by the amount proportional to how powerful Celestial Dusting is and how much the player is moving the character around. Finally rendering the platform the game is run at unstable and could make it eventually crash (considering how persistently the game itself doesn't). I consider this a major bug devs should take a look at. There may be more places in the code that require a proper cleanup of allocated memory that is no longer needed in practice.

I'm surprised unity devs themselves have their engine not sane enough to detect such memory leak instances and use their garbage collectors to solve such issues. If not possible, deploy some sort of code analysis to detect places in the code that may be considered memory unsafe and warn developers before deploying their projects, as much as it is possible to do. Memory leak is a major flaw that not only renders the platform the software is run at unstable but also makes it prone to possible hacking attacks during its instability.

Vampire Survivors - the mysterious level 11586 cap and memory leak.

2024-01-30T23:23:00.016+01:00

Vampire Survivors exp cap? The mysterious level 11586.

On a side not, it was finally requesting 35GB of RAM so had to end the run - the game clearly memory leaked about ~20MB every few seconds when moving and ~5-10% of that when staying still. Still had to wait dozens of minutes after quitting the stage for all the memory swapping and calculations to generate the summary. Afterwards the game wants 7GB of RAM. Celestial Dusting's flowers may not get properly freed from memory when they get off screen before being despawned.

Warning! Looking at max limit broken Celestial Dusting's animation may induce headaches and eye pains even in healthy persons. Would be nice for games to start deploying some sort of sensitization for too fast changing animations (flashing, colors - too rapid changes in displayed image parts to simulate rapid movements or sudden brightness/darkness) as standardized option to check. Said animations in games can go to such extremes that they are hardly neutral to persons with no health issues.

Some sort of lightweight AI that understands limits of what a human can still perceive as an actual motion and, without impacting the game play too much or at all, cleverly optimize such animations to be still understood by the human brain as well as have no negative impact on the person seeing such animations (partial and dynamic motion blurs and smoothed fading in and outs, contouring important elements in the game?)

Alternatively, displays or graphic cards could try to utilize such techniques (as an option and not enforced; possibly protected by a password as a part of parental control), properly process the image to be displayed and, detecting unsafely rapid changing elements of the portion of the screen (could be threshold set based), properly apply various image manipulation techniques to said portions of the screen to protect healthy or otherwise person from suffering from temporary health issues induced by rapid screen changes. Both, as little as possible artifacts as well as health should be taken into consideration.

Improving VDSL speed with Draytek routers.

2022-07-10T13:54:00.022+02:00

Please note: this method is not guaranteed to work, neither guaranteed that DLM won't trigger line banding. It is, however, a method I found working in my case.

First of all. My line is configured as 80/8mbps, however, only 55/7.5mbps is attainable with stock Draytek 2760n Delight settings. It is working in Interleave mode. Said so, currently I'm synced with 59/7.5mbps and I'm still on the way to see if it can be further improved without any physical changes to the line.

Draytek has a console where there is this magical command: vdsl snr [-50~50] with which you can control your downline Signal to Noise Ratio margin. What I see in practice is that when you set it to 50, the modem will refuse to accept signal that is lower than standard SNR +5dB, which may be good to stabilize the line if the standard used by your provider is not enough and you want to manually find the sweet spot before the DLM triggers and possibly cuts the speed more that you could achieve as stable by setting the margin by hand.

My provider wants me to work with 8dB SNR, giving me around 55mbps downline. When I force it with +5dB margin I get 14dB SNR and sync speed around 40mbps. However, the CRC Count and ES Count still shows up with such setting, having absolutely no difference between 8dB and 14dB which makes me believe that the line is not well maintained by our provider here (Orange Polska), not properly shielded or using faulty devices (or shortwaves and other interference surges are so high that it can't be properly avoided), and even with 14dB SNR on 14dB attenuation generates errors and in exactly the same amount as with 8dB SNR.

The good thing is that vdsl snr command allows you to use negative margin and I can confirm it working to some extend on my line. Though, as simple as it is to set positive values, it's not that straight forward when you want DSLAM to give you more. I guarantee you that it doesn't like when you suddenly want to reset the line from your side and negotiate a lower SNR. I'm currently at -1dB offset, though the SNR is still 8dB, my sync downlink speed is higher and offsetting it by -0.1dB did indeed raise it by approx 0.5mbps at a time until I forgot to do it properly and had -0.2dB "stolen" from me.

So how to do it properly?

First you need to remember that your line is under constant measurement in search for errors and link times. Most forums usually define it as 15 min checkpoints and ~24h summary. Thus you SHOULD NOT reset your line too often or preferably at all or you may risk being banded. Said so, there is a bit of "magic" you can perform to reset the line very nicely for DSLAM that I found working well in my case. This method does not necessarily require you to take off the phone line wire and wait 30 min. In fact, disconnecting your phone line from the modem could possibly send some "not nice" noise back to DSLAM that you would like to avoid at all costs, just in case.

In order to see if it is possible to be synced at better speed you need to have a lot of patience and properly measure your line over time (vdsl status) to find the sweet spot when DSLAM "wants" to give you one of the best possible attainable speeds. For me it's 4~6AM my time.

Do NOT use vdsl snr with negative values right away. First use vdsl idle on and wait until your line resets by itself, making your modem quiet so DSLAM can gracefully RESET the line. After this you can either wait 30 minutes or not, it's up to you and maybe how easy it is to trigger line banding with your provider. The next thing is to decrement snr margin by very little, possibly by 1 (0.1dB) at a time, thus vdsl snr -1, vdsl snr -2 and so on and do it preferably every 24h (it can work more often but to avoid banding it is the best to wait between resets). Do not forget vdsl idle on, if you do, it is possible that you won't get any higher speeds and doing it later correctly may even result in DSLAM taking away 0.5mbps from you. You must do vdsl idle on first when reducing SNR margin below 0, acting like a gentleman with DSLAM.

My experience:

- every time I reduced SNR margin by -0.1dB I got +0.5mbps sync

- when I forgot vdsl idle on once I got the same sync with higher attainable speed

- when I corrected my mistake right away I ended up with 0.5mbps being taken from me

- when I yet again (3 time that day, after 2h) reduced SNR by -0.1dB I got back my 0.5mbps

- Result? After forgetting vdsl idle on just once I ended up with -0.2dB margin offset and absolutely no change in actual downlink rate which may be either a limit for me or I triggered DLM to band me a bit, though I was capable of regaining the speed at the loss of 0.2dB of fine-tuning.

Type ? for command help

> vdsl status

  ---------------------- ATU-R Info (hw: annex A, f/w: annex A/B/C) -----------
   Running Mode            :      17A       State                : SHOWTIME
   DS Actual Rate          : 56506000 bps   US Actual Rate       :  7616000 bps
   DS Attainable Rate      : 61409048 bps   US Attainable Rate   :  7897000 bps
   DS Path Mode            :  Interleave    US Path Mode         :  Interleave
   DS Interleave Depth     :      507       US Interleave Depth  :      101
   NE Current Attenuation  :       18 dB    Cur SNR Margin       :        8  dB
   DS actual PSD           :   -10.-5 dB    US actual PSD        :    14. 5  dB
   NE CRC Count            :      176       FE CRC Count         :     9463
   NE ES Count             :       20       FE  ES Count         :     2349
   Xdsl Reset Times        :        0       Xdsl Link  Times     :        8
   ITU Version[0]          : 00000000       ITU Version[1]       : 00000000
   VDSL Firmware Version   : 05-07-09-0F-01-07   [with Vectoring support]
   Power Management Mode   : DSL_G997_PMS_L0
   Test Mode               : DISABLE
  -------------------------------- ATU-C Info ---------------------------------
   Far Current Attenuation :       14 dB    Far SNR Margin       :        8  dB
   CO ITU Version[0]       : b5004244       CO ITU Version[1]    : 434da49e
   DSLAM CHIPSET VENDOR    : < BDCM >

> vdsl idle

% Usage : adsl idle [on | tcpmessage | tcpmessage_off]
% DSL is under [DISABLE] test mode.
% DSL debug tool mode is off.

> vdsl idle on

% DSL is under [IDLE/QUIET] test mode.
% DSL debug tool mode is off.

> vdsl snr -3

VDSL SNR update successfully !
Restarting modem ...

> vdsl status

  ---------------------- ATU-R Info (hw: annex A, f/w: annex A/B/C) -----------
   Running Mode            :      17A       State                : SHOWTIME
   DS Actual Rate          : 57186000 bps   US Actual Rate       :  7702000 bps
   DS Attainable Rate      : 61619704 bps   US Attainable Rate   :  7931000 bps
   DS Path Mode            :  Interleave    US Path Mode         :  Interleave
   DS Interleave Depth     :      513       US Interleave Depth  :      101
   NE Current Attenuation  :       18 dB    Cur SNR Margin       :        8  dB
   DS actual PSD           :   -10.-5 dB    US actual PSD        :    14. 5  dB
   NE CRC Count            :        0       FE CRC Count         :     9463
   NE ES Count             :        0       FE  ES Count         :     2349
   Xdsl Reset Times        :        0       Xdsl Link  Times     :        9
   ITU Version[0]          : 00000000       ITU Version[1]       : 00000000
   VDSL Firmware Version   : 05-07-09-0F-01-07   [with Vectoring support]
   Power Management Mode   : DSL_G997_PMS_L0
   Test Mode               : DISABLE
  -------------------------------- ATU-C Info ---------------------------------
   Far Current Attenuation :       14 dB    Far SNR Margin       :        8  dB
   CO ITU Version[0]       : 00000000       CO ITU Version[1]    : 00000000
   DSLAM CHIPSET VENDOR    : < ----- >

The following day:

> vdsl status

  ---------------------- ATU-R Info (hw: annex A, f/w: annex A/B/C) -----------
   Running Mode            :      17A       State                : SHOWTIME
   DS Actual Rate          : 57186000 bps   US Actual Rate       :  7702000 bps
   DS Attainable Rate      : 61563928 bps   US Attainable Rate   :  7931000 bps
   DS Path Mode            :  Interleave    US Path Mode         :  Interleave
   DS Interleave Depth     :      513       US Interleave Depth  :      101
   NE Current Attenuation  :       18 dB    Cur SNR Margin       :        8  dB
   DS actual PSD           :   -10.-5 dB    US actual PSD        :    14. 5  dB
   NE CRC Count            :       46       FE CRC Count         :     9475
   NE ES Count             :        8       FE  ES Count         :     2354
   Xdsl Reset Times        :        0       Xdsl Link  Times     :        9
   ITU Version[0]          : 00000000       ITU Version[1]       : 00000000
   VDSL Firmware Version   : 05-07-09-0F-01-07   [with Vectoring support]
   Power Management Mode   : DSL_G997_PMS_L0
   Test Mode               : DISABLE
  -------------------------------- ATU-C Info ---------------------------------
   Far Current Attenuation :       14 dB    Far SNR Margin       :        8  dB
   CO ITU Version[0]       : b5004244       CO ITU Version[1]    : 434da49e
   DSLAM CHIPSET VENDOR    : < BDCM >

> vdsl idle on

% DSL is under [IDLE/QUIET] test mode.
% DSL debug tool mode is off.

> vdsl snr -4

VDSL SNR update successfully !
Restarting modem ...

> vdsl status

  ---------------------- ATU-R Info (hw: annex A, f/w: annex A/B/C) -----------
   Running Mode            :      17A       State                : SHOWTIME
   DS Actual Rate          : 57631000 bps   US Actual Rate       :  7748000 bps
   DS Attainable Rate      : 61981992 bps   US Attainable Rate   :  8040000 bps
   DS Path Mode            :  Interleave    US Path Mode         :  Interleave
   DS Interleave Depth     :      517       US Interleave Depth  :      101
   NE Current Attenuation  :       18 dB    Cur SNR Margin       :        8  dB
   DS actual PSD           :   -10.-5 dB    US actual PSD        :    14. 5  dB
   NE CRC Count            :        0       FE CRC Count         :     9484
   NE ES Count             :        0       FE  ES Count         :     2355
   Xdsl Reset Times        :        0       Xdsl Link  Times     :       10
   ITU Version[0]          : 00000000       ITU Version[1]       : 00000000
   VDSL Firmware Version   : 05-07-09-0F-01-07   [with Vectoring support]
   Power Management Mode   : DSL_G997_PMS_L0
   Test Mode               : DISABLE
  -------------------------------- ATU-C Info ---------------------------------
   Far Current Attenuation :       14 dB    Far SNR Margin       :        8  dB
   CO ITU Version[0]       : 00000000       CO ITU Version[1]    : 00000000
   DSLAM CHIPSET VENDOR    : < ----- >

SNR margin +4dB:

SNR margin -0.1dB:

SNR margin -1dB:

Your millage may differ.

Python script to produce images from your commands: vdsl showbins and vdsl showbins up. Must be text file bins_dl.txt and bins_ul.txt respectively. Both starting with bin 0, 1, 2, 3 and ending with 4092, 4093, 4094, 4095. No enter allowed at the end of file. Saves three files (check line with filenames list variable). Labels may not be visible due to adjusted fit to image size - change plt.subplot_adjust numeric parameters if you prefer it otherwise.

import matplotlib.pyplot as plt

def loaddata(file):
    with open(file,mode="r") as f:
        data = f.read()
        data = data.replace("*","\n")
        bins_raw = data.split("\n")
        all_bins = []
        for bin in bins_raw:
            bin_data = bin.split()
            all_bins.append(bin_data)
    bins = []
    bits = []
    snrs = []
    gains = []
    for bin in all_bins:
        bins.append(int(bin[0]))
        bits.append(int(bin[3]))
        snrs.append(int(bin[1]))
        gains.append(int(bin[2]))
    return bins,bits,snrs,gains

data_dl = loaddata("bins_dl.txt")
data_ul = loaddata("bins_ul.txt")

ylabels = ["Bits", "SNR dB", "Gain .1dB"]
filenames = ["dsl_bits_per_bin.png","dsl_snr_per_bin.png","dsl_gain_per_bin.png"]

for x in range(0,3):
    plt.clf()
    plt.xlabel("Bins")
    plt.ylabel(ylabels[x])
    plt.bar(data_dl[0],data_dl[x+1], label="Downstream")
    plt.bar(data_ul[0],data_ul[x+1], label="Upstream")
    plt.subplots_adjust(bottom=0.05, left=0.07, right=1, top=1)
    plt.legend()
    plt.savefig(filenames[x],dpi=1200)

Commands, bins dump and image generating python could possibly be fully automated on Linux machine, using ssh for controlling and getting data from router. Including finding the best possible time to gracefully reset connection for best sync speeds at 24h+ interval.

AMD Ryzens: prototype kind of CPUs?

2022-05-03T17:40:00.000+02:00

Could AMD still be selling their CPUs in not completely finalized form, as kind of still under development chips, or even, Ryzen still being kind of prototype kind of CPU by itself?

Basing on my experience of two out of three Ryzens I own having problems, at least for me, in my personal opinion, AMD sold me an unfinished product. And basing on the problems I had with the latest (4800H), I can believe that it was in fact in some prototype kind of stage still, not thoroughly tested, not a final product that should be put on the market.

What AMD CPUs do I own as products and their problems (if any):

Ryzen 1600X - was struggling a lot to stabilize this CPU as much as possible, still resulting in occasional crashes of applications every few days and BSOD every 1 month or more. 1xxx series had problems with RAM controller, and even though I changed RAM for AMD certified one, which helped too, the instability is still there, no matter what.
Ryzen 2500U - this CPU has no problems so far. Maybe because its clock is pretty low and voltages don't jump as high as on high-end models. My family relative has the same CPU, no problems. So far this is the only Ryzen that is working 100% stable for me without any tinkering.
Ryzen 4800H - this CPU has weird problems. The problems are not noticeable at first until... you run tensorflow in training + sampling mode. After around 24h (shortest was 2h, longest was few days) the notebook will simply crash, usually on "hardware" issue. It does not matter if you force the TDP to 20W, it will still crash.

I could understand 1600X still having problems, it was 1xxx series. A brand new line of products that company should test well but somehow overlooked certain issues just to push it to the market.

I cannot, however, understand that my 4800H is still causing problems and of different kind than 1600X. Isn't it 3 series ahead for someone to get their act together and test the CPU properly and thoroughly in various kind of applications and, if possible, fix the problems or clearly state what the CPU may fail on?

So, how about I clarify what causes my 4800H to fail exactly? I tried to pin-point the problem and I found a solution for this CPU to run completely stable without any BSOD whatsoever.

There are two solutions and are related to Windows power scheme:

Disable Aggressive boost - if you are fine with your CPU running at 1.8Ghz just to be stable, go for that option, or,
Disable Idle - if you are fine with your CPU not saving power when not used and run at full speed instead, with no hint of instability at the same time, go for this one. However, if you overload your CPU your system will be kind of less responsible in this mode (beat me why...), but at the same time some applications may run faster (I got 10-15% speed up with AI training just by disabling Idle mode when the power draw was still 35W)

Unfortunately I have no bios update to maybe mitigate this problem, nor it offers me any kind of tinkering with processor states to try pin-pointing it even further. Nonetheless, I opted for disabling Idle state and controlling my CPU TDP with Renoir Mobile Tuning as otherwise I am at risk of losing my unsaved work.

I'm not the only one having problems with their CPUs, they are more people like this. Whenever there are certain parts being problematic, clearly AMD should run more auditing on that matter to make sure no bad quality CPUs reach the market. Until then you may try BIOS update or tinkering with your power schemes to turn off certain CPU functionalities to make it more if not completely stable.

Will GPT-2 Large (1.5B) run on NVIDIA RTX 3060 12GB? (sampling)

2022-04-22T07:24:00.015+02:00

Yes. The GPT-2 large model with 1.5B parameters will run on RTX 3060 12GB but it barely fits. It will, however, not load in tensorflow <2.4 because you need CUDA 11, at least with this card. Anything below will simply make your GPU visible to tensorflow but it will stay hanged during model loading (showing very slight GPU activity).

It will neither load on tensorflow >2.4, most likely to either tensorflow or CUDA 11.2 taking a bit more memory, hence the model will not fit on the GPU and instead crash with OOM every time.

So what do you need?

Have RTX 3060 as secondary GPU so that it has as much spare memory as possible,
Have drivers compatible with CUDA 11 and cuDNN 8.0 (I used 473.04),
Use tensorflow version 2.4.x only,
Slightly update the code if you want to have vanilla setup,
Do not output too much tokens at the same time,
Keep output amount of tokens low (like 20), reduce to 10 when input >750 tokens.
(optional) Keep looping output->input in an [input size]=[maximum length]-[# token to output] manner while keeping last tokens in input and removing <|endoftext|> token in output to produce one "multi-sampled" long output. For example, for 10 output tokens the input should be the last 1014 tokens, then remove 10 tokens from the beginning and append output at the end of input, and repeat. (tested with TOP-K 1400, temp: 0.95; other settings may require further token reduction; speed: 1T/~0.45s)

Not yet known:

Will it crash during model loading at times due to different seeds?

Deep learing and sampling comparison RTX 1660 vs RTX 3060 with GPT-2 (original code and modified nshepperd implementation):

The deep learning is a bit faster for 3060 but sampling is slower than 1660. However, 1660 barely has 774M fit into it and crashes during model loading sometimes. On 1660 sampling 20T with 774M model took 3~4s where on 3060 it took 5~6s. 3060 has 3% GPU Core utilization during that time which makes it clear that increased amount of CUDA cores does not help in the matter of sampling, in fact slows it down. Most likely due to increase in performance (syntetic benchmark) being ~1.4x overall and ~1.7x for computing, where the amount of CUDA cores increased ~2.4x when comparing 1660 to 3060.

Why is sampling not utilizing neither of cards fully?

I believe it is due to the fact that GPT-2 has auto-regressive decoder which does really decode 1 token at a time, so all those fancy mathematics are simply not enough to fully utilize the GPU card and batching may speed up the process. However, if the model barely fits the batching is a problem as well...

In conclusion. As long as usage of GPT-2 is concerned. If you want a bit faster deep learning and want to fit largest GPT-2 model into GPU memory, upgrading from 1660 to 3060 seems like a valid choice. However, if you're fine with 334M or 774M sampling on GPU with the second occasionally crashing, it may be better to stay with 1660, as each of its CUDA cores seem faster to get smaller sampling times.

Setup:

Ryzen 1600X, 32GB RAM
NVIDIA 3060 12GB, LHR model (secondary card, not used for screen)
Windows 7 x64 (that's right, still ol' good Windows, neither Windows 10 nor Linux)
Python 3.8.10 for Windows
Tensorflow 2.4.4
Latest drivers for Windows 7 x64: 473.04
CUDA11, cuDNN 8.0
PATH with additional data: [Python location] and [Python location]\Scripts; [cuDNN 8.0 location]\bin, [CUDA 11.0 location]\bin, [CUDA 11.0 location]\libnvvp, [CUDA 11.0 location]\extras\CUPTI\libx64 and [CUDA 11.0 location]\include
Modified vanilla GPT-2 code for webpage interface using database, IN->OUT->IN multiple mini-sample looping and some post-processing for longer sentences, eot token removal and simple repeated text removal.
Modified nshepperd's code for deep learning with learning rate updating fixed (original code has it not working at all - it updates it for graph only, not for ADAM), removal of dangerous graph updates slowing down learning process and overloading the memory, saving ADAM state and ability to remove it later, and some other improvements to try replicating similar learning process to that described by GPT creators including warm up, attention drop out and cosine schedule. Deep learning tests were performed on custom parameter, newly initialized models after tokenization. (the fixed code is not available anywhere, as it is not cleaned up and neither proven to not contain other bugs - tough as far as tested it works properly) Why not other code, like huggingface? Because we are going vanilla here by having codes of those who were first to have them working, just fixing and updating them to meet our needs.

WARNING! The text below is the continuation generated by AI for demonstration purposes:

FORMAT STRUCTURE

// The structure of models is common to simple's and deep which we don't power yet, we just use some state holder of basic ideas like // whether we are learning epoch or not, etc, should already be present anyway, that contains the features used // for training, and contains some training properties like block size, re-training state and have min_lr init rate // it holds input, output and samples from training model, and needs to be initialized to some states for test as well // initial: initial learning rate = 0.1, re-training state = 0, k-samples or max_features = 10 // given_state: example_state_list has 50 items, when we have only few samples for LSTM operations we break it into smaller // lists, so that it doesn't hold too many states and may be more compact for training, otherwise for the input we // need a model with only 100 items in params . final: static Random r = new Random(16384); min_lr = 0.005; std::vector<Int32> params_c; std::vector<float> samples_c; Tensor product_tensor = {0}; // y_test: x_train and y_test will be constant tensors, typically float32s thresh = 0.001; // trained gradient_y_updated is the model matrix of (row H, column L) that will be updated during training // // initialize_state: it contains initial model state . final: [[layer_init;_final]] = || {} .final: //[evaluate_grad_y_1_test;_final]] = && { for i := 0 ;i < 5000 ;i++ { r = r.next(); // find the lud of r // skip if there is no different for running tests in test mode, otherwise we don't support // multi-dimensional contributions r.get(); if ((r.get() != r.h) && (r.get() != r.l)) { // we have reported that the current test fails, return the error r = r.queable(); // we only ask if the initial test result is less than a regular failure per test // check whether we get a [positive or not] warning, otherwise don't do anything, if // there is one in the test mode do nothing else if (r.name().empty()) continue; // print out the current state r.get(); // print the current version, and assign a temporary value r} // print out all errors printf("Error : %s the model was already initialized

", r.get()->get_previous()->get_error()); // add before the error buffer if (r.get(), r.get(), r.get() += 100 - 100) { r.get(); // print the current state before the error r.get(); // print out the current version, and assign a temporary value r} // print out the current state ... // and return the tentative error buffer - our error initializer run_err() { // although the error message is usually a single line, one error message is enough, // just to show the user a few messages, so it will be easier see the error later // we need a simple way to move the error left if at all possible r.reformat(); // discard some bad old errors ... printf("%s(%x) ", r.error(), r.err_count()); // add before the static buffer, very important nheq(r); // RNN operation over or over, to move the error forward or backward so it doesn't break the flow r = RNN(DataFrameOperation(params_c, r, 90, 0, 0, samples_c, samples_c, 0) + weights_c); // initialize our state .final: do_run(DataFrameOperation); while(r.get() != r.h) { r.get(); } // if all failures were recoverable, stop looping, we won't generate any inputs .... -1: rnt = r.queue_permutation(); -1: for (;;) { // traverse over all the successes [(max_train_y_original + 1) % max_chance_expected] for i := 0 ;i < max_chance_expected ;i++ ) // assess whether we extracted all the observations from the batch data of the model, // every difficulty, one, in the list if there are more than one observation from our batch, // it must be to set up for more information, and later evaluate the model `input` // predictions or `output` , i am only testing for removal { if (p.metrics() == see_all_see_all) break; // there are enough observations left not to extract more, you can wait up to // the next set of output models for extracting all observations // logistic regression, we want to make sure that no [repeated] observations will occur for this step - .lstm().logistic { // if there is multiple observations, that we saw earlier, we can find the earlier

Real Time Corruptor automation?

2020-06-26T21:40:00.002+02:00

Corruptions, corruptions, corruptions. They can happen everywhere at the most critical moment, like copying important financial data from one place to another. But with backups we are safe, yay! But this is not about backups...

So what is Real Time Corruptor (RTC for short)? It's a piece of software that hooks into an emulator's* memory and modifies random bits to some other bits making the game run in unstable way and, in the worst case scenario, crash completely.

This is how it looks like:

But it's not about RTC either, it's about how you could possibly automate (or integrate) it with something that is not just "Start Auto-Corrupt". Well, with the magic of programming you can.

Lately I wrote a small console APP in C that can run desired intensity level by running this APP with a numeric parameter, being that intensity. "What for, you still need to run it manually," you may ask. Well, that's not really the case if you develop something else as well that can run shell APPs. For that Python is helpful. Without much of a hassle one can create Python script to run that application on a command, but what for? Python can be integrated in a lot of tools... *cough* streaming software *cough*... and then controlled through those means for something outside triggering it to happen - there are those sensor things to developer boards. How about reading loudness and putting that as a corruption. What? You can do that? Yes!

Unfortunately both tools (the APP and the script) are currently closed-source and private, because they were developed with some purpose in mind (no, I'm not revealing it at this point). Buy hey, I can guarantee you that I had it working as a proof-of-concept. Don't believe me? Fine.

And no, I used plain C, not Autohotkey (hint, hint).

* - note that emulation in some countries may be banned. Always use the title you owned and the hardware you owned if you really need to do that (there is just no other way to test how game acts during corruption process)

The text below was generated by AI (GPT-2 1.5B). All information is false and is purely for demonstration purposes:

/* * This program is free software: you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation, either version 3 of the License, or * (at your option) any later version. * a * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program. If not, see <http://www.gnu.org/licenses/>. */ import sys, os import colour char = 0x00 char_buffer = [ 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 ] GPIO_MODE_INT = 1 GPIO_MODE_DMA = 4 GPIO_MODE_HAS_PRIV = 5 GPIO_MODE_VCTRL2 = 6 GPIO_MODE_VAL3 = 7 GPIO_MODE_PRIV = 8 GPIO_MODE_SLEEP = 9 GPIO_MODE_DMA = 10 GPIO_MODE_DIRMA_VCTRL = 11 GPIO_MODE_MAX28 = 12 GPIO_MODE_MIN2 = 13 GPIO_MODE_JOYB3 = 14 GPIO_MODE_LIBRARY = 15 16 /* Power off, mode and mode_reserved space for arbitrary purposes */ 17 GPIO_UNUSED_MAX = GPIO_UNUSED_MAX + 1 18 SET_RBUFSIZ 8000000 19 GPIO_KEY = 0x00 20 GPIO_UNUSED = GPIO_UNUSED - 1 21 GPIO_USE_UBOOK = 0x01 22 SET_RBUFSIZ 3000000 23 GPIO_UNUSED_0 = GPIO_UNUSED_0 + ( 1 << 13 ) 24 GPIO_UNUSED_1 = GPIO_UNUSED_1 + ( 1 << 12 ) 25 GPIO_UNUSED_2 = GPIO_UNUSED_2 + ( 1 << 11 ) 26 GPIO_UNUSED_3 = GPIO_UNUSED_3 + ( 1 << 10 ) 27 GPIO_UNUSED_4 = GPIO_UNUSED_4 + ( 1 << 9 ) 28 GPIO_UNUSED_5 = GPIO_UNUSED_5 + ( 1 << 8 ) 29 GPIO_UNUSED_6 = GPIO_UNUSED_6 + ( 1 << 7 ) 30 GPIO_UNUSED_7 = GPIO_UNUSED_7 + ( 1 << 6 ) 31 GPIO_UNUSED_8 = GPIO_UNUSED_8 + ( 1 << 5 ) 32 GPIO_UNUSED_9 = GPIO_UNUSED_9 + ( 1 << 4 ) 33 GPIO_UNUSED_0_EN = GPIO_UNUSED_0_EN + ( 1 << 3 ) 34 GPIO_UNUSED_1_EN = GPIO_UNUSED_1_EN + ( 1 << 2 ) 35 GPIO_UNUSED_2_EN = GPIO_UNUSED_2_EN + ( 1 << 1 ) 36 GPIO_UNUSED_3_EN = GPIO_UNUSED_3_EN + ( 1 << 0 ) 37 GPIO_UNUSED_4_EN = GPIO_UNUSED_4_EN + ( 1 << 0 ) 38 GPIO_UNUSED_5_EN = GPIO_UNUSED_5_EN + ( 1 << 0 ) 39 GPIO_UNUSED_6_EN = GPIO_UNUSED_6_EN + ( 1 << 0 ) 40 GPIO_UNUSED_7_EN = GPIO_UNUSED_7_EN + ( 1 << 0 ) 41 GPIO_UNUSED_8_EN = GPIO_UNUSED_8_EN + ( 1 << 0 ) 42 GPIO_UNUSED_9_EN = GPIO_UNUSED_9_EN + ( 1 << 0 ) 43 44 HID_KEY = 0x00 45 SET_RBUFSIZ 20 46 KEY_LENASELEN = 7 47 GPIO_EPROM_PORT = 0xA 48 49 50 51 52

Editing AHX music

2020-06-26T21:10:00.004+02:00

This is what editing music in AHX format looks like. AHX are small files created in this tracker that gives the tune this unique chiptune vibe to it. In this tracker, however, you can define multiple notes and effects as an instrument itselfl, making those chiptune-like vibrations just by using the final, desired notes in tracker later without any other effects added (if you wish to).

If you want to try it yourself you would need an Amiga computer with hard drive installed or means to run it in software.

1. You need the proper Amiga running for this, I used Amiga 1200,

2. Connect hard drive to it and boot Workbench 3.0 floppy disk installing it on your hard drive,

3. Download AYS-A233.LHA from ahx and extract the archive,

4. Copy extracted files to your Amiga hard drive,

5. When in workbench, open the hard drive, pick view all files from menu (just browse a bit, you will find it),

6. Browse to the directory with AHX installed and double-click AHX-68000 (or AHX-68020 if you have an Amiga with suitable CPU for that),

7. You should be ready to go.

Remember that right mouse click shows menu on Amiga computers.

Sorry for not being detailed enough, I wanted to keep it brief, so you try and figure it out. I think it's always good to learn by trying things by yourself too, not just doing step-by-step instructions.

Also a small update, I minimized the amount of ADs to a minimum, because I hate those lagging sites that even on my Ryzen work like piece of junk... I hope it's OK to keep at least a bit of that content, right? We have those distractors everywhere nowadays... Oh, look... ADD... :)

Allow as low as 1 voice in CoolSoft VirtualMidi Synth

2018-09-10T18:55:00.001+02:00

Get rid of CoolSoft VirtualMIDI Synth's hardcoded minimum of 16 voices limit.

So I was previously enjoying the CoolSoft VirtualMIDI ini file hack to allow lower than 16 voices to be set, however, in newest version such ability was completely removed by checking the limit in DLL file as well and overriding the ini set value (unlike previous versions). And I don't want to use outdated version just for this. So I went for an older DLL version which was 32-bit but kept overriding my beautiful voices value of 4, used IDA to show me some assembly, searched for voices and guessed the place (dword and 10h, which means 16, doh). Found it in hex editor and remembered the hex string of that assembly, the most important was 00 00 10 00 00 00, as with x64 assembly is a bit different. Downloaded latest version. But hey, there were a bit more than one 00 00 10 00 00 00, oh well... what's the maximum voices limit? 100 000, so now 100 000 in hexadecimal is? 01 86 A0! Now let's reverse it to have it little endian way. Now let's see if we get 00 00 10 00 00 00 and 00 00 A0 86 01 00 close to each other. Yup, of course there is such a place. And this place, in both EXE and DLL is for voices limit. Vua la, program fixed.

To allow as low as 1 voice in virtualmidi synth do the following:

For version 2.5.3 x64:

- get yourself a hex editor,

- make sure VirtualMIDI Synth is closed and no program is using MIDI (like chrome, steam, any music player) and that VirtualMIDI Synth program is closed,

- navigate to your DRIVE:\Program Files\VirtualMIDISynth,

- edit VirtualMIDISynth.exe in your favourite hex editor,

- change byte 10 [16] at position 3B33 [15 155] to 1 [1],

- navigate to DRIVE:\Windows\system32,

- edit VirtualMIDISynth.dll,

- change byte 10 [16] at position 2507A [151 674] to 1 [1],

- reopen VirtualMIDI Synth and change number of voices to 4 and use chiptune soundfont,

- ????

- enjoy 4 voice channels emulation.

Teenage Engineering Pocket Operator PO-20 Arcade

2017-12-16T14:16:00.001+01:00

Small, portable, mobile protracker-like synthesizer PO-20 Arcade with line-in, line-out (headphones), built-in speaker and display powered by 2 AAA batteries.

more demonstration videos at the bottom of this post

This little thing is amazing. It let's You create some cool beats even with Arcade version which is purely designed for chiptunes music. You have 2 octaves for most samples, some are 0-100 finetune range based instead - You control it with left knob. With right knob You alter the filter so the sound is more deep or fading into higher frequencies (kind of like flange effect). You can also do lead in real-time but I'm mostly programming all the patterns as it's very hard to rotate the knob just on the right note - this is unfortunately a con in this model that You can't use buttons in Note mode. Other models have this feature. You can program max 128-chain of patterns and chords which allows to achieve around 6-7 minutes of programmed music as long as You stick to low BPM like 60 and use 120 BPM beats. The max BPM is 240. A limit of 16 positions per pattern is a bit low but then You can use step multiplier which allows You to pick 2, 3, 4, 6 and 8 multiplier modes for a note witch can quite well overcome this limit. You can also easily copy a pattern so adding new instrument lines is very easy. Unfortunately You can't connect it to PC through USB port or similar to transfer Your creation in ProTracker-like form so if You want to keep the chiptune in mod format You would pretty much need to recreate it there. Nonetheless the device allows You to feel how the ProTracker-like music can be creater by showing You how to arrange your notes to play accordingly to given chord.

The device features:

a clock
an alarm
2 octaves (A-1 to A-3)
0-100 filter effect per note
16 samples
16 patterns
16 chords
16 live effects
16 volume settings
128-chains
parameter locking
a drone sound
fade out
tempo and swing
step multiplier
copying and clearing a pattern
various sync modes with left-right audio channel splitting
built-in speaker (I didn't know about that before buying! Nice feature!)
powered by 2 AAA batteries (mine rechargeable usually last 2-3 days)

Here are some examples of the music I was able to create by purely improvising:

House:

Rave:

Standard chiptune-like loop:

Techno Club:

Different bpm-swing variation of the above video, for demonstration purposes:

60 bpm setting (120 real) / 80 swing

65 bpm setting (130 real) / 50 swing

90 bpm setting (180 real) / 50 swing

If you want to acquire this device (or a set of them) check out the product page: https://teenageengineering.com/products/po. They sure let you forget about the world around you when you step into music creation mode. :)

If you live in Poland or countries nearby, at this moment it's best to buy it from Czech distributor Kytary: https://kytary.pl/Search/?term=pocket%20operator. They offer reasonable prices and ship with DHL (select PPL as transportation method) and allow paying with Paypal.

7-ZIP finetuner

2017-01-28T22:46:00.002+01:00

Small application to try various 7-ZIP settings to find best combination for given input data. This usually results in smaller archive size.

Help:

7-ZIP Finetuner v1.00 by Mr_KrzYch00

This program will test various 7-ZIP switches known to improve

compression ratio. Smallest resulting file will be kept on disk.

Usage: 7zft [switches] [input files ...] [archive name]

Switches:

/l# - 7z executable (default: "c:\program files\7-zip\7z.exe")

/d# - dictionary size (default: 512m)

/s# - solid block size (default: on)

/m# - match finder cycles (default: 4294967295)

This program is provided as-is, use at Your own risk.

Support the author: https://www.paypal.me/MrKrzYch00

Optimization sample:

f:\MyScripts\7z test>7zft Ht_edf_v1 Ht_edf_v1

7-ZIP Finetuner v1.00 by Mr_KrzYch00

LC: 3, LP: 0, PB: 2, YX: 5, FB: 273 (INITIAL RUN) - best: 1372745

LC: 1, LP: 0, PB: 2, YX: 5, FB: 273 - best: 1372697 (-48)

LC: 2, LP: 0, PB: 2, YX: 5, FB: 273 - best: 1372483 (-214)

LC: 1, LP: 1, PB: 2, YX: 5, FB: 273 - best: 1372441 (-42)

LC: 0, LP: 2, PB: 2, YX: 5, FB: 273 - best: 1372264 (-177)

LC: 0, LP: 0, PB: 3, YX: 5, FB: 273 - best: 1372248 (-16)

LC: 1, LP: 0, PB: 3, YX: 5, FB: 273 - best: 1372061 (-187)

LC: 1, LP: 1, PB: 3, YX: 5, FB: 273 - best: 1371695 (-366)

LC: 0, LP: 1, PB: 4, YX: 5, FB: 273 - best: 1371445 (-250)

LC: 1, LP: 1, PB: 4, YX: 5, FB: 273 - best: 1371404 (-41)

LC: 1, LP: 1, PB: 4, YX: 5, FB: 272 - best: 1371393 (-11)

LC: 1, LP: 1, PB: 4, YX: 5, FB: 269 - best: 1371342 (-51)

LC: 1, LP: 1, PB: 4, YX: 5, FB: 260 - best: 1371305 (-37)

LC: 1, LP: 1, PB: 4, YX: 5, FB: 253 - best: 1371303 (-2)

LC: 1, LP: 1, PB: 4, YX: 5, FB: 252 - best: 1371115 (-188)

LC: 1, LP: 1, PB: 4, YX: 5, FB: 172 - best: 1371084 (-31)

LC: 1, LP: 1, PB: 4, YX: 5, FB: 155 - best: 1371069 (-15)

LC: 1, LP: 1, PB: 4, YX: 5, FB: 141 - best: 1371010 (-59)

LC: 1, LP: 1, PB: 4, YX: 5, FB: 134 - best: 1370992 (-18)

LC: 1, LP: 1, PB: 4, YX: 5, FB: 125 - best: 1370980 (-12)

LC: 1, LP: 1, PB: 4, YX: 5, FB: 121 - best: 1370964 (-16)

LC: 1, LP: 1, PB: 4, YX: 5, FB: 70 - best: 1370908 (-56)

LC: 1, LP: 1, PB: 4, YX: 5, FB: 68 - best: 1370846 (-62)

LC: 1, LP: 1, PB: 4, YX: 5, FB: 63 - best: 1370812 (-34)

LC: 1, LP: 1, PB: 4, YX: 5, FB: 54 - best: 1370781 (-31)

LC: 1, LP: 1, PB: 4, YX: 5, FB: 52 - best: 1370756 (-25)

LC: 1, LP: 1, PB: 4, YX: 5, FB: 5

SUMMARY:

- Size: 1370756

- LC: 1

- LP: 1

- PB: 4

- Method: LZMA2

- YX: 5

- FB: 52

You can download it from: http://virtual.4my.eu

Zopfli on Odroid XU4

2017-01-10T15:58:00.003+01:00

Lately I got myself Odroid XU4 microcomputer based on Exynoss 5422 octa core (4x 2Ghz + 4x 1.4Ghz). Surprisingly it can run all 8 cores simultaneously. Unfortunately its size is a bit bigger than Odroid U3 so if You made Yourself some custom made multi-odroid chassis, You would need to rebuild it if You plan to put XU4 in there as well. I mainly bought it to have headless OS with Zopfli KrzYmod running on, doing 999,999 iterations per block. I can confirm that it's around 50% faster than Odroid U3, with single threaded zopfli compression. In my small test of compressing a file using multi-threaded compression it was 55% faster than Odroid U3 O/Ced at 1.92Ghz (limited to 6 threads and various block sizes so it couldn't show its full potential, which I would believe could be up to 75% faster).

Unfortunately it's getting up to 95*C temperature when using all 4 big cores + 1-2 small cores, when small cores are not used the temperature goes to 93*C max. I'm now waiting for Grizzly Kryonaut thermal paste to arrive to apply it on all my RPis and Odroids (and later to apply better cooling solutions in case that won't help much) to make sure they won't get throttled much, especially during summer here.

There is also the same problem that occurs on Odroid U3 being SIGSEGVs with zopfli when long running certain blocks of data. The kernel seems newer on Odroid XU4, so I'm still confused by this error as it is not occurring with x86/x64 builds, might be GCC to generate bogus compilation, or one of the special switches I pass to it for faster builds.

Some tests:

                |       ORIGINAL -O2                   |  Mr_KrzYch00's Zopfli KrzYmod  |
       X        |-----------------------------------------------------------------------|
                | 2016.04.20 | 2016.05.19 | Makefile*  | v16.5.22 --t0 | v16.5.22 --t99 |
----------------------------------------------------------------------------------------|
           real | 15m35.201s | 14m6.836s  | 11m21.864s | 10m56.287s    | 7m11.551s      | - ARM Cortex-A9 - quad @ 1.92Ghz
Odroid U3  user | 14m40.965s | 14m6.480s  | 11m21.580s | 10m55.970s    | 19m30.025s     | (quad-core)
NEON       sys  | 0m53.925s  | 0m0.125s   | 0m0.095s   | 0m0.135s      | 0m0.805s       |
----------------------------------------------------------------------------------------|
           real |            |            |            | 7m21.517s     | 4m40.129s      | - ARM Cortex-A7 - quad @ 1.4Ghz
Odroid XU4 user | [missing]  | [missing]  | [missing]  | 7m21.345s     | 8m27.750s      | - ARM Cortex-A15 - quad @ 2.0Ghz
NEON+VFPV4 sys  |            |            |            | 0m0.080s      | 0m0.180s       | (octa-core)
----------------------------------------------------------------------------------------|
           real | 7m41.104s  | 5m50.010s  | 4m31.952s  | 4m1.706s      | 2m35.052s      | - Core i7-3630QM @ 2.7Ghz
Fedora x86 user | 7m20.247s  | 5m39.968s  | 4m24.474s  | 3m53.982s     | 6m27.671s      | (quad core)
rawhideAVX sys  | 0m9.306s   | 0m0.242s   | 0m0.071s   | 0m0.268s      | 0m2.843s       |
----------------------------------------------------------------------------------------|
           real | 5m59.811s  | 4m25.060s  | 3m38.561s  | 3m31.905s     | 2m17.897s      |
Fedora x64 user | 5m56.214s  | 4m24.860s  | 3m38.295s  | 3m31.776s     | 5m57.761s      |
rawhideAVX sys  | 0m3.543s   | 0m0.118s   | 0m0.208s   | 0m0.056s      | 0m1.636s       |


* - same original Zopfli 2016.05.19 with Zopfli KrzYmod's makefile + profile guided optimizations:
        https://github.com/MrKrzYch00/zopfli/blob/master/Makefile
  --t99 is to force all blocks compression at once - 6 threads used

Fedora rawhide kernel panic Virtualbox

2016-12-02T14:09:00.002+01:00

I'm now running on a very slow PC being disassembled Asus 1215N with O/Ced Atom CPU to 2.35Ghz. As far as I managed to upgrade windows 7 to x64 version I can not make magically VT-x appear on this CPU, that was the reason why I ran into problems when tried to boot Fedora rawhide i686 in Virtualbox. Basically the problem occurs with kernels versioned 4.7+, it seems to boot fine with 4.6 one. On Virtualbox forums there was some discussion that it may be related to virtual box display driver from virtualbox extension pack being installed (and of course VT-x being unavailable making I/O APIC option not performing correctly without it - or so I believe after I did my tests). So far I have it running with kernel 4.6 and dnf-ing around the OS, that takes ages due to slowness caused by no VT-x being available. If I find a way on how to run kernel 4.9 with my configuration, I will share this information.

And here is some interesting stuff with DNF, it can even erase "None", lol:

Chrome/Chromium hangs often

2016-12-01T18:25:00.001+01:00

Recently I ran into a problem when my Chromium browser was hanging a lot when I activated the window or few seconds after opening new tab. First I VACUUM-ed every SQLite file in my profile directory... Didn't help... Then I run procmon tool to show me what exactly was it doing on my harddrive, then I found out it was reading some files, so I went through every one of them, renaming them by adding .bak extension. And then vua la, I found out what was causing it. It turns out it was Preferences file growing too big (even though it was just 450KB) when You visited a lot of various sites in the past, as it saves some preferences for every one of them and the engine/parts of code to read it back and analyze seem to be very inefficient, not to say, shitty. So imo that part requires some kind of inspection to be performed by Chromium/Chrome DEVs as currently it's giving Chromium even 1 minute hangs every now and then.

To fix Chrome/Chromium hangs:

- close all running instances of Chrome/Chromium,

- go to Your Chrome/Chromium Default profile, usually %LocalAppData%\Chromium\User Data\Default for Windows users,

- find Preferences file,

- rename it to Preferences.bak or remove it,
- run Chrome/Chromium,
- lags/hangs should be gone now,
- in case they are not, You may be having some other problematic file that, maybe, uses JSON format to store data as well. In other words its structure should be having a lot of { } " " : chars as well, maybe the other one got too big for You?

You may have some settings reset to default like Chrome/Chromium asking to be default browser or having Translate messages scaring You a lot, unfortunately, You must get past it again now. :)

Analysis and conversion of binary files with the use of PHP CLI

2016-10-03T16:28:00.002+02:00

Last time I gave myself a mission to translate some game to Polish language. These aren't ordinary text files but instead custom scripts binary coded. At the bottom of file there is a lot of letters in a row. You can't practically tell which part of the text will be displayed in the exact moment in the game. By deep analysis of such files You can come to some conclusions. By viewing one of them in hexadecimal editor You can notice that some characters repeat over and over again. You can notice that they occur every 4 bytes, so these are most likely binary coded 32-bit variables (long int). However, as it is a script that hold a lot of numbers, then You can imply, that the big part of them will be of no use for You. Also by carefully looking at it You can notice repeating [FF] bytes one next to another. So these will be most likely numeric variables represented as signed type (positive & negative), because I doubt that the software the company used to edit these scripts required anyone to type 4 billion-like numbers, these were most likely negative numbers close to zero like -1 or -3.

Why would You need to know all of this though?
Such detailed knowledge about these files is needed to have full control over the length of text You wish to modify. As I previously wrote, such text doesn't have a visible split points, in which part of the game which part of the text is shown. So something else must used for game to tell the beginning at which to start loading the data and it's length.
Let's start from the beginning of file. You can see a DCPB tag there which identifies file type used by game. Next You can try to decode 4 bytes after that, because it may be used to represent an offset or certain data length in this file. So [BC] [BA] [01] [00] is: 188 + 186 * 256 + 1 * 256^2 + 0 * 256^3 (little endian) = 113 340. Now You can check the file size, it is 117 150 bytes. So exactly how I wrote, it must represent a length of some data in this file. So now You can check if something interesting is located at the decimal address of 113 348 in this file. [64] [00] [00] [00] is located there, in other words the value of 100. But what it actually means? An offset? A value? Something else? If this is an offset then for sure at the distance of +/- 100 bytes nothing interesting exists. Let's go to 4 bytes ahead then. [20] [03] [00] [00] is located there, in other words the value of 800. Let's see if something interesting exists 800 bytes ahead of this place. Bull's eye! By adding 800 to current offset, from which we read this value, first text character exists. So You can be almost 100% sure that this is the offset of first part of the text which will be displayed somewhere in the game. Now You can go 800 bytes back and read next 4 bytes as signed long int. [12] [00] [00] [00] is located there, the value of 18. Is it maybe 800 + 18 as a some kind of offset? Hmm. Let's check another 4 bytes ahead, [32] [03] [00] [00], the value of 818. It looks like that in exactly THIS PLACE the offset really is coded. So what exactly means the value of 18 we previously decoded? 800 + 18 = 818! If You assume that 800 and 818 were offsets, then what occurs after ever of them (in this case the value of 18) is most likely the length (the number of bytes) of the text that will be read by the game. Now You can count how many occurrences of such 8 byte ranges exist before You reach the place where the text resides. It looks like there is 100 occurrences of them, so 4 bytes coded before the first offset ([64] [00] [00] [00] = 100) most likely indicates the amount of lines of the text to translate.

Let's begin?
After getting more or less accustomed to file structure, which You are interested in, You can translating the text. But let's think for a bit. If You have 100 occurrences, then every of them needs to have offset and length properly aligned. Won't there be a problem if the first part of the text gets a bit longer than the original? It is known that other languages than the original, the game was transcribed in, may result in bigger amount of translation data. You would then need to manually alter every next offset! Huge amount of work and a lot of time required. Is there maybe some other way to do this?

PHP CLI scripts as a helpful tool to convert files to more user friendly version.
Easiest way is to "convert" the file in such a way that You will have the text exactly split line by line, You won't be bothered by some kind of offsets, and You can skip directly to translating the text. You must remember, though, that such file must be properly converted back to its previous format, including the changes to binary data that points the offsets and lengths of parts of the text/lines. You can attain much more when programming in C, for example by directly reading the file and dynamically creating text input boxes for text inputting/altering basing on original game script. However, it can take more time to do it, it all depends if You have parts of code to use from Your other projects, or not.

Analysis and conversion of files with the use of PHP scripts.
You can write PHP scripts by checking the manual and using plain notepad. First You would need to create an algorithm or have the ability to do it on-fly in Your head. You must have in mind the analysis You did before. Let's try to more or less describe how the file structure looks like:
String: "DCPB",
4 bytes of signed long int: data length, which we You won't modify,
data, which You won't modify,
4 bytes of signed long int: amount of text occurrences,
{4 bytes of signed long int: nth text offset, 4 bytes of signed long int: nth text length} * amount of occurrences,
text as a whole.

By looking at the above You would for sure need a loop/function to read 4 bytes and translate them to long int. For sure You will also need to read a parts of data which You won't modify, and as such, they would need to be coded in a way that it will be safe to text edit such file. I advice to use base64_encode, and to apply deflate compression before it:
base64_encode(gzcompress($buf,9));
This way You will achieve a file to edit which will be much smaller that the original and it won't hold a lot of numbers and letters You are not interested in at the beinning (base64).
When You will be in a place to read offsets and lengths of the text, then for sure You will need dynamic arrays, which will hold these offsets and lengths in 2 arrays (or 2-dimensional array under one variable, which one You prefer most). In PHP it's relatively easy, in C You would need to malloc/realloc a lot, and to remember to NULL byte at the end of array of characters etc. Dynamic arrays are best created here by loop repeating [text occurence amount] times by reading 4 bytes two times, decoding them and putting them as long ints in 2 array variables, for example:

for($x=0;$x<$amount;++$x) {

$buf = fread($file1,4);

$pos[] = (ord($buf[3]) << 24) + (ord($buf[2]) << 16) + (ord($buf[1]) << 8) + ord($buf[0]) + 12 + $rawdatalength;

$buf = fread($file1,4);

$len[] = (ord($buf[3]) << 24) + (ord($buf[2]) << 16) + (ord($buf[1]) << 8) + ord($buf[0]);

}

Of course You can read whole 4 bytes and use proper bitshifting or other methods. Whichever method suits You best, most important is to do have conversion working without errors. Code performance in usually required in production environment or when a lot of data is concerned.
After the loop finishes You need to start another one, which will use data from previously created arrays, properly read the amount of characters and output it in separate lines into text editing friendly file. So to be sure it's best that You use fseek for offsets and fread to read amount of characters, and as last fwrite. Example:

for($x=0;$x<$amount;++$x) {

fseek($file1,$pos[$x],SEEK_SET);

$text = fread($file1,$len[$x]);

fwrite($file2,$text . PHP_EOL);

}

This is more or less everything You would need to do to create user friendly text editing file. You can of course put an amount of data You got from compressing and base64-coding the data. This way You will avoid the fgets error of 1024 characters limit if You will utilize fgets to read Your text files. In case of the data that You will base64 decode and decompress, You will use fread($file,$amount_of_bytes_to_read).

The converting back to binary script needs to be done in reverse order. For sure You would need to recreate "DCPB" tag and write 4 bytes that You will read from the length of Your decoded, decompressed data, write this data, recreate proper offset and lengths of lines which Your script will read from text file You used to safely translate the game text. After You finish writing the coding part You need to test the original coded file against Your translated one, if the differences between the two start after first offset pointing to the text and if all offsets and lengths properly point to the parts of translated text. If You can confirm that it's exactly as it should be then You succeeded in creating coder/decoder of script files which will make Your translation alteration much easier.

File used as an example: http://virtual.4my.eu/samples/SSS05_01.scripb

Analiza plików binarnych i ich przetwarzanie za pomocą PHP CLI

2016-10-03T15:07:00.002+02:00

Ostatnio podjąłem się misji tłumaczenia pewnej gry na język polski. Nie są to zwykłe pliki tekstowe, a niestandardowe skrypty zakodowane w postaci binarnej. Na samym dole plików znajduje się tekst w jednym ciągu. Nie można tak naprawdę odróżnić, która część tekstu będzie wyświetlana w którym momencie gry. Poprzez dogłębną analizę tychże plików możemy jednak wywnioskować pewne rzeczy. Podglądając jeden z nim w edytorze hexadecymalnym możemy zauważyć pewną powtarzalność występujących w nim znaków. Można dostrzec, że występują one co 4 bajty, czyli prawdopodobnie są to zakodowane liczby 32-bitowych zniennych (long int). Jednakże, skoro jest to skrypt i składa się z samych liczb i jest ich dość wiele, to możemy założyć, że duża ilość z nich nie będzie nam tak naprawdę do niczego potrzebna. Także przyglądając się szczegółowo można zauważyć powtarzające sie bajty [FF] koło siebie, czyli zmienne te będą najprawdopodobniej w trybie signed (dodatnie i ujemne), ponieważ nie wydaje mi się, aby w oprogramowaniu, którego firma używała do edycji tychże skryptów, ktokolwiek wpisywał dodatnie numery rzędu 4 milardów, raczej były to negatywne cyfry takie jak -1 czy -3.

Jednak po co nam ta wiedza?

Takie szczegółowe zapoznanie się z plikiem jest nam potrzebne aby mieć całkowitą kontrolę nad długością tekstu, który chcemy modyfikować. Tak jak wcześniej wspominałem sam tekst na końcu pliku nie ma wyraźnie widocznych podziałów, w którym momencie w grze dana jego część będzie się wyświetlać. W takim razie coś innego musi kontrolować ich początek wczytywania i ilość znaków jaka będzie wyświetlana.

Zacznijmy od początku pliku, widać tam tag DCPB, który identyfikuje rodzaj pliku wykorzystywany przez grę. Następnie spróbujmy zdekodować 4 kolejne bajty, ponieważ może to być oznaczenie offsetu lub rozmiar pewnej ilości danych w tym pliku. Czyli [BC] [BA] [01] [00] to: 188 + 186 * 256 + 1 * 256^2 + 0 * 256^3 (little endian) = 113 340. Teraz spójżmy na rozmiar pliku: 117 150. Czyli tak jak pisałem jest to prawdopodobnie oznaczenie długości pewnej części pliku. Spróbujmy w takim razie zobaczyć czy coś ciakawego znajduję się po przeskoczeniu na adres decymalny 113 348 w pliku. Znajduje się tam [64] [00] [00] [00], czyli wartość 100. Tylko co to może oznaczać? Offset? Wartość? Coś innego? Jeśli jest to offset to na pewno w odległości +/-100 bajtów od tego miejsca nic ciekawe nie znajduje się. W takim razie przeskoczmy 4 bajty do przodu. Znajduje się tam [20] [03] [00] [00], czyli wartość 800, zobaczmy czy w odległości 800 bajtów od tego miejsca znajduje się coś ciekawego. Bingo! Dodając 800 do aktualnego offsetu, z którego odczytaliśmy tę własnie wartość, znajduje się pierwsza litera/znak tekstu. Czyli możemy prawie w 100% potwierdzić, że jest to oznaczenie offsetu pierwszej części teksu, która będzie wyświetlana w grze. Cofnijmy się spowrotem o 800 bajtów i odczytajmy następne 4 bajty jako signed long int. Znajduje się tam [12] [00] [00] [00] czyli wartość 18. Czyżby było to 800 + 18 jako offset? Hmm. Sprawdźmy następne 4 bajty, [32] [03] [00] [00] czyli 818. Wygląda na to, że własnie W TYM miejscu jest tak naprawdę offset. Co w takim razie oznacza ta wcześniejsza liczba 18? 800 + 18 = 818! Jeśli przyjąć, że 800 i 818 to offsety, to w takim razie to co znajduje się po każdym offsecie (w tym przypadku 18) to pewnie jest długość (ilość bajtów) wczytywanego tekstu przez grę w danym miejscu. Czyli każde 8 bajtów jest używane do wczytania parametrów danego tekstu wyświetlanego w grze. W takim razie policzmy ile razy takie 8 bajtów występuje zanim zacznie się tekst. Wygląda na to, że jest 100 wystąpień, czyli 4 bajty zdekodowane przed pierwszym offsetem ([64] [00] [00] [00] = 100) oznaczają najprawdopodobniej ilość linii tekstu do przetłumaczenia.

Zaczynamy?

Znająć już mniej więcej strukturę pliku, która nas interesuje możemy przejść do tłumaczenia tekstu na język polski. Jednak pomyślmy przez chwilę. Jeśli mamy 100 wystąpień, każde z nich musi mieć dopasowany offset i długość to czy nie będzie to dla nas problemem jeśli wydłuży nam się pierwszy tekst przy tłumaczeniu? Wiadomym jest, że w języku polskim BARDZO często tłumaczony tekst jest dłuższy niż w innych językach. Będziemy musieli wtedy ręcznie zmieniać wszystkie offsety! Ogromna robota i duża ilość czasu potrzebna. Czy jest może inny sposób?

Skrypty PHP CLI jako pomocna dłoń przy konwertowaniu plików na bardziej przyjazny dla użytkownika.

Najprostrzym sposobem jest "przerobić" plik w taki sposób abyśmy mieli tekst ładnie podzielony linia po linii i nie przejmowali się jakimiś offsetami tylko samym tłumaczeniem tekstu. Pamiętać jednak należy, że taki plik musi być poprawnie zakodowany spowrotem do poprzedniego formatu wraz z odpowiednim nałożeniem zmian w binarnej części oznaczającej offsety i długości danych części tekstu/linii. Oczywiście piszac w języku C możemy osiągnać znacznie więcej, na przykład bezpośrednio wczytując plik dynamicznie tworzyć pola do wpisywania/zmiany tekstu bazująć na oryginalnym pliku skryptu gry. Może to jednak zająć więcej czasu, zależy czy mamy części kodu z naszych wcześniejszych projektów, które możemy wykorzystać, czy nie.

Analiza i przetwarzanie pliku skryptami PHP.

Skrypty PHP można pisać w zwykłym notatniku korzystając z manuala. Na początek jednak musimy stworzyć sobie algorytm lub mieć zdolność tworzenia ich w locie w naszej głowie. Musimy pamiętać o naszej wcześniejszej analizie. Rozpiszmy sobie jak mniej więcej wygląda struktura pliku:

String: "DCPB",

4 bajty signed long int: długość danych, których nie modyfikujemy,

dane, których nie modyfikujemy,

4 bajty signed long int: ilość wystąpień tekstu,

{4 bajty signed long: offset tekstu n, 4 bajty signed long: długość tesktu n} * ilość wystąpień,

tekst w jednym ciągu.

Patrząc na powyższy podział na pewno będziemy musieli mieć pętle/funkcję wczytywania 4 bajtów i tłumaczenia ich na long int. Na pewno takżę będziemy musieli wczytać część danych, których nie modyfikujemy, a w ziązku z tym, że muszą one być zachowane w trybie bezpiecznym do edycji tekstowej, proponuję wykorzystać base64_encode, a przed tym kompresję deflate:

base64_encode(gzcompress($buf,9));

W ten sposób nasze pliki do edycji będą znaczne mniejsze i nie będą na początku zawierać nadmiernej ilości nieinteresującego nas ciągu cyfr i liter (base64).

Gdy przejdziemy do wczytywania offsetów i dlugości tekstu, to na pewno będziemy potrzebowali dynamiczne arraye, które będą nam przechowywać offsety i długości w 2 tablicach (lub 2-wymiarowej tablicy pod jedną zmienna, jak wolisz). W PHP jest to stosunkowo łatwe, w C byśmy musieli dużo mallocować/reallocować, pamiętać o bajcie NULL na końcu tablicy znaków itp. Dynamiczne arraye najlepiej tworzyć w pętli powtarzającej się [ilość wystąpień tekstu] razy czytając 2 razy po 4 bajty i wrzucając zdekodowane longi do 2 zmienny będących tablicami, na przykład:

for($x=0;$x<$amount;++$x) {

$buf = fread($plik1,4);

$pos[] = (ord($buf[3]) << 24) + (ord($buf[2]) << 16) + (ord($buf[1]) << 8) + ord($buf[0]) + 12 + $rawdatalength;

$buf = fread($plik1,4);

$len[] = (ord($buf[3]) << 24) + (ord($buf[2]) << 16) + (ord($buf[1]) << 8) + ord($buf[0]);

}

Oczywiście można wczytywać całe 8 bajtów i odpowiednio je bitshiftować lub wykorzystać inne metody. To jaką metodę przyjmiesz zależy od Ciebie, najważniejsze jest aby konwertowanie bezbłednie działało, wydajność kodu potrzebna jest najczęściej tylko w środowiskach produkcyjnych lub przy dużej ilości przetwarzanych danych.

Po zakończonej pętli trzeba rozpocząc następną, która będzie korzystać z danych w poprzednio utworzonych tablicach i wczytywać odpowiednią ilość tekstu oraz zapisywać to w odrębnych liniach do pliku przyjaznego do edycji tekstowej. Czyli dla pewności najlepiej wykorzystać fseek do offsetów i fread do długości, a następnie fwrite. Przykład:

for($x=0;$x<$amount;++$x) {

fseek($plik1,$pos[$x],SEEK_SET);

$text = fread($plik1,$len[$x]);

fwrite($plik2,$text . PHP_EOL);

}

To mniej więcej wszystko co trzeba zrobić aby utworzyć plik przyjazny do edycji tekstowej. Możemy oczywiście na początku pliku zapisać jeszcze długość danych jaką osiągnęliśmy kompresująć i base64-kodując te dane. W ten sposób unikniemy problemów z limitem do 1024 znaków jeśli będziemy korzystać z fgets do czytania pliku, który przygotowaliśmy do zakodowania. W przypadku ciągu danych, które będzie dekodować z base64 i dekompresować, będziemy korzystać z fread($plik,$ilosc_bajtow_do_wczytania).

Kodowanie musi odbywać się odwrotnie do dekodowania, na pewno będziemy musieli odtworzyć tag "DCPB" zapisać 4 bajty długości danych poprzez przeliczenie długości zakodowanego, zdekompresowaneego ciągu bajtów, zapisać ten ciąg i odtworzyć poprawnie offsety i długości z linii, które skrypt będzie czytał z pliku tekstowego, który wykorzystywaliśmy do bezpiecznego tłumaczenia gry. Po napisaniu prawidłowo części kodującej należy przetestować oryginalny zakodowany plik i przetłumaczony, czy różnice między oboma plikami zaczynają się po pierwszym offsecie tekstu i czy wszystkie offsety i długości poprawnie wskazują na części przetłumaczonego tekstu. Jeśli tak rzeczywiście jest, to udało nam się poprawnie stworzyć koder/dekoder skryptów gry, w sposób, który znacznie ułatwia nam nanoszenie zmian na tekst wyświetlający się w niej.

Plik wykorzystany jako przykład: http://virtual.4my.eu/samples/SSS05_01.scripb

Zopfli KrzYmod updates

2016-03-16T22:33:00.002+01:00

Zopfli fork also called Zopfli KrzYmod got some updates.

Basically I put there some latest zopfli changes which impact compression, enabled --legacy mode to work with multi-files as well (to provide new Zopfli functionality fully, notably additional splitting last which is unfortunatelly omitted in KrzYmod algorithm with newest changes). Added --brotli switch to use Brotli's Huffman for RLE which provides different block splitting and smaller files at times (on my test case I got 13 bytes reduction on ~2KB gz file of minified JS).

I will put release a bit later after I get rid of all errors, I think there are some with zopflipng crashing or so. You can, however, compile it Yourself from git.

https://github.com/MrKrzYch00/zopfli

Putty Windows x64 and Linux builds (x86, x64, ARMv5, ARMv7)

2016-03-10T22:35:00.002+01:00

If anybody is interested, I just compiled Putty for below systems:

- Windows x64 (CLI & GUI),

- Linux x86 (CLI & GUI-GTK),

- Linux x64 (CLI & GUI-GTK),

- Linux ARMv7 (CLI & GUI-GTK),
- Linux ARMv5 (CLI Only).

All builds feature Linker-time optimizations with Linker Plugin (except ARMv5), are static.
ARMv7 build is NEON FPU optimised, built on Odroid U3.
ARMv5 was built on Zyxel NSA-220 (oarm, uClibc).

CLI & GUI builds contain: fuzzterm, pageant, plink, pscp, psftp, pterm, putty, puttygen, puttytel.
CLI only builds contain: fuzzterm, plink, pscp, psftp, puttygen.

I know it's kind of pointless to have Putty on Linux but hey, maybe someone is so used to operate it on Windows he/she is willing to have it on Linux as well. :)

Windows x86 builds are available on the official site, so I'm not compiling those.

Here it is: http://virtual.4my.eu/putty

Why should You care for x64 builds? Well, if You run x64 system usually x64 builds should be faster on those systems. With putty it should make a difference when using tunnels to transfer large amounts of data or copying files with other tools provided. I didn't make any comparison charts myself, but if You care to do them, You can share the information. :)

KrzYVideoFixer v1.05 is out

2016-03-01T07:21:00.000+01:00

Few hours ago the new version of KrzYVideoFixer is out which in my opinion contains all possible checks agains video/audio stream to get rid of parts with missing video frames without re-encoding. Remember, You need ffprobe and ffmpeg in the same directory or globally accessible directory. You can download both programs from this site. The only thing left to fix is to support filenames with spaces which I am already testing locally and will be available in v1.06. I did not find any other problems in the last few hours. Here is an example of 3 times run (when video is very broken the output file may still be a bit damaged due to good audio stream being sometimes cut out by ffmpeg if it's near damaged fragment resulting in time skips still occuring, usually 2nd run fixes all problems):

e:\VLCPortable\kvf -s12 349834500000.flv

KrzYVideoFixer v1.05 by Mr_KrzYch00

-> Detecting a minimum of 12 missing frames!

|/ Extracting chunk #01: 0 to 3:25:00.7 (L: 3:25:00.7)
|/ Extracting chunk #02: 3:25:11.5 to 3:55:11.1 (L: 0:29:59.6)
!--> Audio unstable at keyframe: 3:55:12.9, trying next
|/ Extracting chunk #03: 3:55:13.9 to 6:05:13.9 (L: 2:10:00.0)
!--> Audio unstable at keyframe: 6:05:16.6, trying next
|/ Extracting chunk #04: 6:05:17.6 to 6:55:00.7 (L: 0:49:43.1)
!--> Audio unstable at keyframe: 7:45:02.3, trying next
|/ Extracting chunk #05: 7:45:03.4 to 7:45:35.3 (L: 0:00:31.9)
?-> Video drop detected at: 7:45:36.8~7:45:42.52, but audio is stable!
!--> Audio unstable at keyframe: 7:45:42.9, trying next
|/ Extracting chunk #06: 7:45:44.0 to 7:55:04.1 (L: 0:09:20.1)
!--> Audio unstable at keyframe: 7:56:08.0, trying next
|/ Extracting chunk #07: 7:56:09.1 to 8:02:44.5 (L: 0:06:35.4)
!--> Audio unstable at keyframe: 8:02:51.3, trying next
|/ Extracting chunk #08: 8:02:52.4 to 8:05:51.1 (L: 0:02:58.7)
!--> Audio unstable at keyframe: 8:05:53.3, trying next
|/ Extracting chunk #09: 8:05:54.4 to 8:08:36.4 (L: 0:02:42.0)
!--> Audio unstable at keyframe: 8:08:43.3, trying next
?-> Video drop detected at:8:08:44.3~8:08:47.05, but audio is stable!
|/ Extracting chunk #10: 8:08:47.2 to 8:35:43.8 (L: 0:26:56.6)
!--> Audio unstable at keyframe: 8:36:06.1, trying next
|/ Extracting chunk #11: 8:36:07.2 to 8:55:09.3 (L: 0:19:02.1)
!--> Audio unstable at keyframe: 8:55:31.4, trying next
|/ Extracting chunk #12: 8:55:32.5 to 8:57:51.8 (L: 0:02:19.3)
!--> Audio unstable at keyframe: 8:58:08.0, trying next
|/ Extracting chunk #13: 8:58:09.4 to 8:58:16.8 (L: 0:00:07.4)
!--> Audio unstable at keyframe: 8:58:39.0, trying next
?-> Video drop detected at:8:58:24.6~8:58:38.99, but audio is stable!
|/ Extracting chunk #14: 8:58:40.2 to 8:58:50.3 (L: 0:00:10.1)
?-> Video drop detected at:8:59:12.7~8:59:18.99, but audio is stable!
!--> Audio unstable at keyframe: 8:59:19.6, trying next
!--> Audio unstable at keyframe: 8:59:36.2, trying next
|/ Extracting chunk #15: 8:59:37.7 to 8:59:41.6 (L: 0:00:03.9)
?-> Video drop detected at:8:59:43.2~8:59:50.99, but audio is stable!
!--> Audio unstable at keyframe: 8:59:51.6, trying next
|/ Extracting chunk #16: 8:59:52.7 to 9:00:07.7 (L: 0:00:15.0)
!--> Audio unstable at keyframe: 9:00:16.6, trying next
!--> Audio unstable at keyframe: 9:00:17.6, trying next
|/ Extracting chunk #17: 9:00:23.5 to 9:00:24.9 (L: 0:00:01.4)
!--> Audio unstable at keyframe: 9:00:40.1, trying next
|/ Extracting chunk #18: 9:00:41.3 to 9:00:50.4 (L: 0:00:09.1)
!--> Audio unstable at keyframe: 9:01:05.6, trying next
|/ Extracting chunk #19: 9:01:06.7 to 9:01:24.8 (L: 0:00:18.1)
?-> Video drop detected at:9:01:36.2~9:01:42.99, but audio is stable!
!--> Audio unstable at keyframe: 9:01:44.2, trying next
|/ Extracting chunk #20: 9:01:45.7 to 9:34:35.7 (L: 0:32:50.0)
?-> Video drop detected at:9:34:37.3~9:34:39.03, but audio is stable!
!--> Audio unstable at keyframe: 9:34:40.0, trying next
?-> Video drop detected at:9:34:43.6~9:34:55.13, but audio is stable!
!--> Audio unstable at keyframe: 9:34:55.6, trying next
?-> Video drop detected at:9:34:59.2~9:35:11.16, but audio is stable!
!--> Audio unstable at keyframe: 9:35:12.0, trying next
?-> Video drop detected at:9:35:15.6~9:35:27.20, but audio is stable!
!--> Audio unstable at keyframe: 9:35:28.0, trying next
!--> Audio unstable at keyframe: 9:35:44.2, trying next
|/ Extracting chunk #21: 9:35:45.4 to 9:35:49.3 (L: 0:00:03.9)
WARNING! Last audio frame>video frame: 9:36:09.91>9:35:59.16, ignoring audio!
WARNING! Last audio frame>video frame: 9:36:09.91>9:36:07.12, ignoring audio!
!--> Audio unstable at keyframe: 9:36:15.1, trying next
|/ Extracting chunk #22: 9:36:16.2 to 9:36:19.7 (L: 0:00:03.5)
?-> Video drop detected at:9:36:21.2~9:36:31.12, but audio is stable!
!--> Audio unstable at keyframe: 9:36:32.0, trying next
|/ Extracting chunk #23: 9:36:33.1 to 9:44:35.5 (L: 0:08:02.4)
?-> Video drop detected at:9:44:37.0~9:44:39.12, but audio is stable!
!--> Audio unstable at keyframe: 9:44:40.1, trying next
|/ Extracting chunk #24: 9:44:41.2 to 9:44:52.4 (L: 0:00:11.2)
!--> Audio unstable at keyframe: 9:45:05.6, trying next
|/ Extracting chunk #25: 9:45:11.8 to 9:45:25.1 (L: 0:00:13.3)
?-> Video drop detected at:9:45:26.7~9:45:35.23, but audio is stable!
!--> Audio unstable at keyframe: 9:45:35.6, trying next
|/ Extracting chunk #26: 9:45:36.7 to 9:45:46.2 (L: 0:00:09.5)
!--> Audio unstable at keyframe: 9:45:55.6, trying next
|/ Extracting chunk #27: 9:45:56.7 to 9:46:09.2 (L: 0:00:12.5)
?-> Video drop detected at:9:46:10.7~9:46:15.15, but audio is stable!
!--> Audio unstable at keyframe: 9:46:16.0, trying next
|/ Extracting chunk #28: 9:46:17.6 to 9:46:19.6 (L: 0:00:02.0)
?-> Video drop detected at:9:46:21.2~9:46:31.14, but audio is stable!
!--> Audio unstable at keyframe: 9:46:31.5, trying next
|/ Extracting chunk #29: 9:46:32.6 to 9:46:35.6 (L: 0:00:03.0)
!--> Audio unstable at keyframe: 9:46:47.5, trying next
!--> Audio unstable at keyframe: 9:46:48.5, trying next
?-> Video drop detected at:9:46:49.5~9:46:56.14, but audio is stable!
|/ Extracting chunk #30: 9:46:56.4 to 9:47:00.0 (L: 0:00:03.6)
!--> Audio unstable at keyframe: 9:47:20.3, trying next
|/ Extracting chunk #31: 9:47:21.7 to 9:47:35.1 (L: 0:00:13.4)
!--> Audio unstable at keyframe: 9:47:44.2, trying next
?-> Video drop detected at:9:51:35.7~9:51:45.34, but audio is stable!
|/ Extracting chunk #32: 9:47:45.3 to 9:51:46.0 (L: 0:04:00.7)
!--> Audio unstable at keyframe: 9:51:51.4, trying next
!--> Audio unstable at keyframe: 9:51:52.5, trying next
!--> Audio unstable at keyframe: 9:51:53.6, trying next
!--> Audio unstable at keyframe: 9:51:59.2, trying next
!--> Audio unstable at keyframe: 9:52:16.3, trying next
WARNING! Last audio frame>video frame: 9:52:27.71>9:52:17.73, ignoring audio!
?-> Video drop detected at:9:52:18.7~9:52:31.17, but audio is stable!
|/ Extracting chunk #33: 9:52:31.4 to 9:52:33.6 (L: 0:00:02.2)
!--> Audio unstable at keyframe: 9:52:50.4, trying next
|/ Extracting chunk #34: 9:52:51.6 to 9:52:58.7 (L: 0:00:07.1)
!--> Audio unstable at keyframe: 9:53:13.8, trying next
?-> Video drop detected at:9:53:18.6~9:53:19.85, but audio is stable!
|/ Extracting chunk #35: 9:53:14.9 to 9:53:19.9 (L: 0:00:05.0)
!--> Audio unstable at keyframe: 9:53:28.1, trying next
!--> Audio unstable at keyframe: 9:53:29.3, trying next
?-> Video drop detected at:9:53:37.2~9:53:51.17, but audio is stable!
|/ Extracting chunk #36: 9:53:35.5 to 9:53:59.3 (L: 0:00:23.8)
!--> Audio unstable at keyframe: 9:54:16.6, trying next
!--> Audio unstable at keyframe: 9:54:35.0, trying next
|/ Extracting chunk #37: 9:54:36.1 to 9:54:39.3 (L: 0:00:03.2)
?-> Video drop detected at:9:54:40.9~9:54:47.16, but audio is stable!
!--> Audio unstable at keyframe: 9:54:47.2, trying next
|/ Extracting chunk #38: 9:54:48.4 to 9:54:49.6 (L: 0:00:01.2)
!--> Audio unstable at keyframe: 9:55:05.8, trying next
|/ Extracting chunk #39: 9:55:07.1 to 9:55:17.3 (L: 0:00:10.2)
!--> Audio unstable at keyframe: 9:55:30.9, trying next
|/ Extracting chunk #40: 9:55:32.0 to 9:55:35.1 (L: 0:00:03.1)
!--> Audio unstable at keyframe: 9:55:46.2, trying next
|/ Extracting chunk #41: 9:55:47.3 to 9:55:49.7 (L: 0:00:02.4)
!--> Audio unstable at keyframe: 9:56:02.0, trying next
|/ Extracting chunk #42: 9:56:03.1 to 11:15:01.7 (L: 1:18:58.6)
!--> Audio unstable at keyframe: 11:15:18.5, trying next
|/ Extracting chunk #43: 11:15:19.6 to 11:35:20.2 (L: 0:20:00.6)
!--> Audio unstable at keyframe: 11:35:22.0, trying next
|/ Extracting chunk #44: 11:35:23.1 to 12:21:03.8 (L: 0:45:40.7)
!--> Audio unstable at keyframe: 12:21:16.7, trying next
|/ Extracting chunk #45: 12:21:17.8 to 13:46:58.6 (L: 1:25:40.8)
!--> Audio unstable at keyframe: 13:47:05.0, trying next
!--> Audio unstable at keyframe: 13:47:06.0, trying next
?-> Video drop detected at:13:47:17.9~13:47:19.32, but audio is stable!
|/ Extracting chunk #46: 13:47:12.1 to 13:47:22.6 (L: 0:00:10.5)
!--> Audio unstable at keyframe: 13:47:35.4, trying next
!--> Audio unstable at keyframe: 13:47:36.4, trying next
WARNING! Last audio frame>video frame: 13:47:41.16>13:47:38.40, ignoring audio!
?-> Video drop detected at:13:47:38.9~13:47:43.32, but audio is stable!
!--> Audio unstable at keyframe: 13:47:43.8, trying next
|/ Extracting chunk #47: 13:47:44.9 to 13:47:48.2 (L: 0:00:03.3)
!--> Audio unstable at keyframe: 13:48:00.7, trying next
?-> Video drop detected at:13:48:03.8~13:48:07.66, but audio is stable!
|/ Extracting chunk #48: 13:48:01.8 to 14:13:28.1 (L: 0:25:26.3)
!--> Audio unstable at keyframe: 14:13:40.4, trying next
|/ Extracting chunk #49: 14:13:41.5 to 15:00:24.7 (L: 0:46:43.2)
?-> Video drop detected at:15:00:26.2~15:00:31.50, but audio is stable!
!--> Audio unstable at keyframe: 15:00:32.2, trying next
|/ Extracting chunk #50: 15:00:33.3 to 15:06:18.7 (L: 0:05:45.4)
?-> Video drop detected at:15:06:20.3~15:06:23.55, but audio is stable!
!--> Audio unstable at keyframe: 15:06:24.4, trying next
|/ Extracting chunk #51: 15:06:25.5 to 15:10:10.8 (L: 0:03:45.3)
!--> Audio unstable at keyframe: 15:10:17.5, trying next
|/ Extracting chunk #52: 15:10:18.6 to 15:14:45.4 (L: 0:04:26.8)
?-> Video drop detected at:15:14:47.1~15:14:55.61, but audio is stable!
!--> Audio unstable at keyframe: 15:14:56.1, trying next
|/ Extracting chunk #53: 15:14:57.2 to 15:15:53.9 (L: 0:00:56.7)
!--> Audio unstable at keyframe: 15:16:01.1, trying next
|/ Extracting chunk #54: 15:16:02.2 to 15:16:58.5 (L: 0:00:56.3)
!--> Audio unstable at keyframe: 15:17:05.6, trying next
|/ Extracting chunk #55: 15:17:06.7 to 15:18:03.5 (L: 0:00:56.8)
!--> Audio unstable at keyframe: 15:18:11.1, trying next
|/ Extracting chunk #56: 15:18:12.2 to 15:19:50.3 (L: 0:01:38.1)
!--> Audio unstable at keyframe: 15:20:02.9, trying next
|/ Extracting chunk #57: 15:20:04.0 to 15:22:18.0 (L: 0:02:14.0)
!--> Audio unstable at keyframe: 15:22:24.9, trying next
|/ Extracting chunk #58: 15:22:26.0 to 15:28:46.2 (L: 0:06:20.2)
!--> Audio unstable at keyframe: 15:28:58.7, trying next
|/ Extracting chunk #59: 15:28:59.8 to 15:29:40.1 (L: 0:00:40.3)
!--> Audio unstable at keyframe: 15:29:46.6, trying next
|/ Extracting chunk #60: 15:29:47.7 to 15:30:05.5 (L: 0:00:17.8)
!--> Audio unstable at keyframe: 15:30:18.1, trying next
|/ Extracting chunk #61: 15:30:19.2 to 15:45:35.0 (L: 0:15:15.8)
?-> Video drop detected at:15:45:36.5~15:45:43.73, but audio is stable!
!--> Audio unstable at keyframe: 15:45:44.3, trying next
|/ Extracting chunk #62: 15:45:45.4 to 15:54:40.7 (L: 0:08:55.3)
!--> Audio unstable at keyframe: 15:54:48.7, trying next
|/ Extracting chunk #63: 15:54:49.8 to 15:55:10.8 (L: 0:00:21.0)
!--> Audio unstable at keyframe: 15:55:23.1, trying next
|/ Extracting chunk #64: 15:55:24.2 to 15:58:54.5 (L: 0:03:30.3)
!--> Audio unstable at keyframe: 15:59:04.8, trying next
|/ Extracting chunk #65: 15:59:05.9 to 16:04:16.3 (L: 0:05:10.4)
!--> Audio unstable at keyframe: 16:04:28.7, trying next
|/ Extracting chunk #66: 16:04:29.8 to 16:07:06.1 (L: 0:02:36.3)
!--> Audio unstable at keyframe: 16:07:12.8, trying next
|/ Extracting chunk #67: 16:07:13.9 to 16:07:20.9 (L: 0:00:07.0)
?-> Video drop detected at:16:07:22.5~16:07:27.76, but audio is stable!
!--> Audio unstable at keyframe: 16:07:28.0, trying next
?-> Video drop detected at:16:14:15.0~16:14:15.76, but audio is stable!
|/ Extracting chunk #68: 16:07:29.1 to 16:19:24.1 (L: 0:11:55.0)
!--> Audio unstable at keyframe: 16:19:27.8, trying next
?-> Video drop detected at:16:19:25.7~16:19:27.80, but audio is stable!
|/ Extracting chunk #69: 16:19:28.9 to 16:20:54.7 (L: 0:01:25.8)
!--> Audio unstable at keyframe: 16:21:07.2, trying next
|/ Extracting chunk #70: 16:21:08.3 to 16:30:39.6 (L: 0:09:31.3)
!--> Audio unstable at keyframe: 16:30:51.7, trying next
|/ Extracting chunk #71: 16:30:52.8 to 16:33:13.0 (L: 0:02:20.2)
?-> Video drop detected at:16:33:14.6~16:33:19.74, but audio is stable!
!--> Audio unstable at keyframe: 16:33:19.9, trying next
|/ Extracting chunk #72: 16:33:21.0 to 16:39:35.9 (L: 0:06:14.9)
?-> Video drop detected at:16:39:37.5~16:39:43.80, but audio is stable!
!--> Audio unstable at keyframe: 16:39:44.2, trying next
|/ Extracting chunk #73: 16:39:45.3 to 16:41:08.4 (L: 0:01:23.1)
?-> Video drop detected at:16:41:10.0~16:41:11.80, but audio is stable!
!--> Audio unstable at keyframe: 16:41:11.9, trying next
|/ Extracting chunk #74: 16:41:13.0 to 16:46:07.4 (L: 0:04:54.4)
?-> Video drop detected at:16:46:09.0~16:46:15.79, but audio is stable!
!--> Audio unstable at keyframe: 16:46:16.6, trying next
|/ Extracting chunk #75: 16:46:17.7 to 16:47:43.0 (L: 0:01:25.3)
!--> Audio unstable at keyframe: 16:47:55.8, trying next
|/ Extracting chunk #76: 16:47:56.9 to 16:48:51.6 (L: 0:00:54.7)
!--> Audio unstable at keyframe: 16:48:58.4, trying next
|/ Extracting chunk #77: 16:48:59.5 to 16:52:54.2 (L: 0:03:54.7)
!--> Audio unstable at keyframe: 16:53:06.5, trying next
?-> Video drop detected at:16:53:07.9~16:53:11.88, but audio is stable!
|/ Extracting chunk #78: 16:53:07.6 to 16:54:02.4 (L: 0:00:54.8)
?-> Video drop detected at:16:54:04.0~16:54:07.88, but audio is stable!
!--> Audio unstable at keyframe: 16:54:08.2, trying next
|/ Extracting chunk #79: 16:54:09.3 to 17:00:57.1 (L: 0:06:47.8)
?-> Video drop detected at:17:00:58.6~17:01:03.88, but audio is stable!
!--> Audio unstable at keyframe: 17:01:04.5, trying next
|/ Extracting chunk #80: 17:01:05.6 to 17:04:42.5 (L: 0:03:36.9)
?-> Video drop detected at:17:04:44.1~17:04:47.88, but audio is stable!
!--> Audio unstable at keyframe: 17:04:48.2, trying next
|/ Extracting chunk #81: 17:04:49.3 to 17:06:20.5 (L: 0:01:31.2)
?-> Video drop detected at:17:06:22.1~17:06:23.88, but audio is stable!
!--> Audio unstable at keyframe: 17:06:24.0, trying next
|/ Extracting chunk #82: 17:06:25.1 to 17:10:49.0 (L: 0:04:23.9)
?-> Video drop detected at:17:10:50.6~17:10:55.88, but audio is stable!
!--> Audio unstable at keyframe: 17:10:56.2, trying next
|/ Extracting chunk #83: 17:10:57.3 to 17:16:30.5 (L: 0:05:33.2)
?-> Video drop detected at:17:16:32.1~17:16:39.88, but audio is stable!
!--> Audio unstable at keyframe: 17:16:40.0, trying next
|/ Extracting chunk #84: 17:16:41.1 to 17:19:11.4 (L: 0:02:30.3)
!--> Audio unstable at keyframe: 17:19:23.9, trying next
|/ Extracting chunk #85: 17:19:25.0 to 17:23:25.8 (L: 0:04:00.8)
!--> Audio unstable at keyframe: 17:23:37.3, trying next
|/ Extracting chunk #86: 17:23:38.4 to 17:27:21.2 (L: 0:03:42.8)
?-> Video drop detected at:17:27:22.7~17:27:27.86, but audio is stable!
!--> Audio unstable at keyframe: 17:27:28.5, trying next
|/ Extracting chunk #87: 17:27:29.6 to 17:31:26.7 (L: 0:03:57.1)
?-> Video drop detected at:17:31:28.2~17:31:35.88, but audio is stable!
!--> Audio unstable at keyframe: 17:31:36.1, trying next
|/ Extracting chunk #88: 17:31:37.2 to 17:32:38.3 (L: 0:01:01.1)
!--> Audio unstable at keyframe: 17:32:51.0, trying next
|/ Extracting chunk #89: 17:32:52.1 to 17:35:00.0 (L: 0:02:07.9)
!--> Audio unstable at keyframe: 17:35:07.2, trying next
|/ Extracting chunk #90: 17:35:08.3 to 17:41:43.1 (L: 0:06:34.8)
!--> Audio unstable at keyframe: 17:41:55.4, trying next
|/ Extracting chunk #91: 17:41:56.5 to 17:43:28.4 (L: 0:01:31.9)
?-> Video drop detected at:17:43:30.0~17:43:35.88, but audio is stable!
!--> Audio unstable at keyframe: 17:43:36.2, trying next
|/ Extracting chunk #92: 17:43:37.3 to 17:46:40.4 (L: 0:03:03.1)
!--> Audio unstable at keyframe: 17:46:52.2, trying next
|/ Extracting chunk #93: 17:46:53.3 to 17:47:29.7 (L: 0:00:36.4)
?-> Video drop detected at:17:47:31.2~17:47:35.92, but audio is stable!
!--> Audio unstable at keyframe: 17:47:36.8, trying next
|/ Extracting chunk #94: 17:47:37.9 to 17:54:43.5 (L: 0:07:05.6)
!--> Audio unstable at keyframe: 17:54:50.7, trying next
|/ Extracting chunk #95: 17:54:51.8 to 17:59:36.1 (L: 0:04:44.3)
!--> Audio unstable at keyframe: 17:59:47.9, trying next
|/ Extracting chunk #96: 17:59:49.0 to 18:01:19.3 (L: 0:01:30.3)
!--> Audio unstable at keyframe: 18:01:31.9, trying next
|/ Extracting chunk #97: 18:01:33.0 to 18:02:16.2 (L: 0:00:43.2)
?-> Video drop detected at:18:02:17.8~18:02:23.92, but audio is stable!
!--> Audio unstable at keyframe: 18:02:24.0, trying next
|/ Extracting chunk #98: 18:02:25.1 to 18:02:42.2 (L: 0:00:17.1)
?-> Video drop detected at:18:02:43.7~18:02:47.92, but audio is stable!
!--> Audio unstable at keyframe: 18:02:48.2, trying next
|/ Extracting chunk #99: 18:02:49.3 to 18:03:06.3 (L: 0:00:17.0)
?-> Video drop detected at:18:03:07.9~18:03:11.92, but audio is stable!
!--> Audio unstable at keyframe: 18:03:12.2, trying next
|/ Extracting chunk #100: 18:03:13.3 to 18:03:40.4 (L: 0:00:27.1)
!--> Audio unstable at keyframe: 18:03:47.0, trying next
|/ Extracting chunk #101: 18:03:48.1 to 18:06:50.2 (L: 0:03:02.1)
!--> Audio unstable at keyframe: 18:06:56.6, trying next
|/ Extracting chunk #102: 18:06:57.7 to 18:13:48.7 (L: 0:06:51.0)
!--> Audio unstable at keyframe: 18:13:55.3, trying next
?-> Video drop detected at:18:15:03.1~18:15:03.96, but audio is stable!
|/ Extracting chunk #103: 18:13:56.4 to 18:17:24.5 (L: 0:03:28.1)
!--> Audio unstable at keyframe: 18:17:31.9, trying next
|/ Extracting chunk #104: 18:17:33.0 to 18:17:52.3 (L: 0:00:19.3)
!--> Audio unstable at keyframe: 18:18:05.0, trying next
|/ Extracting chunk #105: 18:18:06.1 to 18:20:23.3 (L: 0:02:17.2)
!--> Audio unstable at keyframe: 18:20:33.0, trying next
|/ Extracting chunk #106: 18:20:34.1 to 18:20:44.1 (L: 0:00:10.0)
!--> Audio unstable at keyframe: 18:20:51.2, trying next
|/ Extracting chunk #107: 18:20:52.3 to 18:24:00.2 (L: 0:03:07.9)
!--> Audio unstable at keyframe: 18:24:12.3, trying next
|/ Extracting chunk #108: 18:24:13.4 to 18:24:32.3 (L: 0:00:18.9)
!--> Audio unstable at keyframe: 18:24:44.3, trying next
|/ Extracting chunk #109: 18:24:45.4 to 18:36:20.7 (L: 0:11:35.3)
!--> Audio unstable at keyframe: 18:36:24.0, trying next
?-> Video drop detected at:18:36:22.3~18:36:23.95, but audio is stable!
|/ Extracting chunk #110: 18:36:25.1 to 18:36:53.3 (L: 0:00:28.2)
!--> Audio unstable at keyframe: 18:36:56.0, trying next
?-> Video drop detected at:18:36:54.9~18:36:55.95, but audio is stable!
|/ Extracting chunk #111: 18:36:57.1 to 18:37:57.9 (L: 0:01:00.8)
!--> Audio unstable at keyframe: 18:38:10.4, trying next
|/ Extracting chunk #112: 18:38:11.5 to 18:39:18.8 (L: 0:01:07.3)
!--> Audio unstable at keyframe: 18:39:28.9, trying next
|/ Extracting chunk #113: 18:39:30.0 to 18:46:52.8 (L: 0:07:22.8)
!--> Audio unstable at keyframe: 18:46:59.5, trying next
|/ Extracting chunk #114: 18:47:00.6 to 18:49:44.4 (L: 0:02:43.8)
?-> Video drop detected at:18:49:46.0~18:49:51.93, but audio is stable!
!--> Audio unstable at keyframe: 18:49:52.4, trying next
?-> Video drop detected at:18:54:39.0~18:54:39.93, but audio is stable!
WARNING! pts/dts N/A at frames: 18:59:07.9, 18:59:07.9
|/ Extracting chunk #115: 18:49:53.5 to 18:59:07.9 (L: 0:09:14.4)

|/ Merging chunks -> 349834500000.flv__done__.mp4

The Operation Completed Successfully!

|/ Waiting for audio scan to finish . . .

e:\VLCPortable>kvf -s12 349834500000.flv__done__.mp4

KrzYVideoFixer v1.05 by Mr_KrzYch00

-> Detecting a minimum of 12 missing frames!

?-> Video drop detected at:7:16:52.9~7:16:55.59, but audio is stable!
|/ Extracting chunk #01: 0 to 8:52:21.7 (L: 8:52:21.7)
!--> Audio unstable at keyframe: 8:52:31.5, trying next
!--> Audio unstable at keyframe: 8:52:32.2, trying next
|/ Extracting chunk #02: 8:52:33.4 to 8:52:45.5 (L: 0:00:12.1)
!--> Audio unstable at keyframe: 8:52:47.1, trying next
!--> Audio unstable at keyframe: 8:52:47.2, trying next
!--> Audio unstable at keyframe: 8:53:03.0, trying next
?-> Video drop detected at:8:52:51.0~8:53:02.95, but audio is stable!
?-> Video drop detected at:12:43:59.6~12:44:01.04, but audio is stable!
?-> Video drop detected at:12:44:09.9~12:44:13.81, but audio is stable!
?-> Video drop detected at:15:07:00.4~15:07:01.25, but audio is stable!
?-> Video drop detected at:15:44:16.2~15:44:20.19, but audio is stable!
?-> Audio drop detected at:16:54:20.6, but video is stable!
?-> Video drop detected at:17:02:18.6~17:02:19.43, but audio is stable!
?-> Video drop detected at:17:39:59.2~17:40:00.16, but audio is stable!
WARNING! pts/dts N/A at frames: 17:44:28.1, 17:44:28.1
|/ Extracting chunk #03: 8:53:04.1 to 17:44:28.1 (L: 8:51:24.0)

|/ Merging chunks -> 349834500000.flv__done__.mp4__done__.mp4

The Operation Completed Successfully!

|/ Waiting for audio scan to finish . . .

e:\VLCPortable>kvf -s12 349834500000.flv__done__.mp4__done__.mp4

KrzYVideoFixer v1.05 by Mr_KrzYch00

-> Detecting a minimum of 12 missing frames!

?-> Video drop detected at:7:16:52.9~7:16:55.59, but audio is stable!
?-> Video drop detected at:12:43:29.8~12:43:31.20, but audio is stable!
?-> Video drop detected at:12:43:40.1~12:43:43.97, but audio is stable!
?-> Video drop detected at:15:06:30.6~15:06:31.41, but audio is stable!
?-> Video drop detected at:15:43:46.4~15:43:50.35, but audio is stable!
?-> Audio drop detected at:16:53:50.7, but video is stable!
?-> Video drop detected at:17:01:48.8~17:01:49.59, but audio is stable!
?-> Video drop detected at:17:39:29.4~17:39:30.32, but audio is stable!
WARNING! pts/dts N/A at frames: 17:43:58.2, 17:43:58.3
Video file is not damaged!

The Operation Completed Successfully!

|/ Waiting for audio scan to finish . . .

e:\VLCPortable>

KrzYVideoFixer v1.04 will be delayed

2016-02-28T03:07:00.002+01:00

As the title says, v1.04 will be a bit delayed... This is because it turns out there are a bit more factors to take care about when "chunking", "merging" the video in order to get rid of all the damage without re-encoding. If You are interested on what my kvf console output is at the moment when checking very damaged video stream, check below:

KrzYVideoFixer v1.04 by Mr_KrzYch00

-> Saving output file when input is not damaged!

-> Detecting a minimum of 10 missing frames!

WARNING! pts/dts mismatch at frame: 0:38:28.08
WARNING! pts/dts mismatch at frame: 0:38:28.12
|/ Extracting chunk #01: 0 to 0:38:29.5 (L: 0:38:29.5)
WARNING! pts/dts mismatch at frame: 1:21:34.08
WARNING! pts/dts mismatch at frame: 1:21:34.12
|/ Extracting chunk #02: 0:38:33.7 to 1:21:34.0 (L: 0:43:00.3)
!--> Audio unstable at keyframe: 1:21:39.4, trying next
WARNING! pts/dts mismatch at frame: 1:24:04.12
WARNING! pts/dts mismatch at frame: 1:24:04.24
|/ Extracting chunk #03: 1:21:40.5 to 1:24:04.1 (L: 0:02:23.6)
?-> Video drop detected at: 1:24:05.6~1:24:08.64, but audio is stable!
!--> Audio unstable at keyframe: 1:24:09.4, trying next
?-> Video drop detected at: 1:24:13.9~1:24:16.64, but audio is stable!
WARNING! pts/dts mismatch at frame: 1:24:18.64
WARNING! pts/dts mismatch at frame: 1:24:18.68
?-> Video drop detected at: 1:24:18.7~1:24:26.48, but audio is stable!
?-> Video drop detected at: 1:24:27.2~1:24:32.64, but audio is stable!
WARNING! pts/dts mismatch at frame: 3:17:08.80
WARNING! pts/dts mismatch at frame: 3:17:08.84
|/ Extracting chunk #04: 1:24:10.5 to 3:17:08.7 (L: 1:52:58.2)
!--> Audio unstable at keyframe: 3:17:15.9, trying next
WARNING! pts/dts mismatch at frame: 3:42:35.64
WARNING! pts/dts mismatch at frame: 3:42:35.68
|/ Extracting chunk #05: 3:17:17.0 to 3:42:35.6 (L: 0:25:18.6)
!--> Audio unstable at keyframe: 3:42:43.0, trying next
!--> Audio unstable at keyframe: 3:42:44.0, trying next
WARNING! pts/dts mismatch at frame: 3:42:44.08
WARNING! pts/dts mismatch at frame: 3:42:44.12
?-> Video drop detected at: 3:42:44.1~3:42:45.00, but audio is stable!
?-> Video drop detected at: 3:42:45.6~3:42:48.64, but audio is stable!
WARNING! pts/dts mismatch at frame: 3:44:57.36
WARNING! pts/dts mismatch at frame: 3:44:57.40
?-> Video drop detected at: 3:44:57.4~3:44:58.32, but audio is stable!
?-> Video drop detected at: 3:44:59.1~3:45:05.92, but audio is stable!
?-> Video drop detected at: 3:53:35.9~3:53:36.92, but audio is stable!
WARNING! pts/dts mismatch at frame: 3:53:40.00
WARNING! pts/dts mismatch at frame: 3:53:40.04
|/ Extracting chunk #06: 3:42:45.1 to 3:53:39.9 (L: 0:10:54.8)
?-> Video drop detected at: 3:53:41.6~3:53:44.92, but audio is stable!
!--> Audio unstable at keyframe: 3:53:45.4, trying next
?-> Video drop detected at: 3:53:50.1~3:53:52.92, but audio is stable!
WARNING! pts/dts mismatch at frame: 3:53:57.16
WARNING! pts/dts mismatch at frame: 3:53:57.28
|/ Extracting chunk #07: 3:53:46.5 to 3:53:57.2 (L: 0:00:10.7)
!--> Audio unstable at keyframe: 3:54:04.4, trying next
?-> Video drop detected at: 3:54:06.0~3:54:08.92, but audio is stable!
WARNING! pts/dts mismatch at frame: 4:08:27.56
WARNING! pts/dts mismatch at frame: 4:08:27.60
|/ Extracting chunk #08: 3:54:05.5 to 4:08:27.5 (L: 0:14:22.0)
?-> Video drop detected at: 4:08:29.0~4:08:32.92, but audio is stable!
!--> Audio unstable at keyframe: 4:08:33.1, trying next
WARNING! pts/dts mismatch at frame: 4:10:29.40
WARNING! pts/dts mismatch at frame: 4:10:29.44
|/ Extracting chunk #09: 4:08:34.2 to 4:10:29.3 (L: 0:01:55.1)
!--> Audio unstable at keyframe: 4:10:31.1, trying next
WARNING! pts/dts mismatch at frame: 4:40:53.40
WARNING! pts/dts mismatch at frame: 4:40:53.44
/ Video drop detected, Scanning Audio: 4:21:04.19

KrzYVideoFixer & mf2t/t2mf (linux native & armv7, windows)

2016-02-24T23:58:00.002+01:00

In the last few weeks I was playing with downloading some publically available video streams in order to upload them to Youtube for better awareness and easy of access. The one thing that bothered me the most was that Youtube converter was getting stuck at some of them (the common infinite 95% processing bug or processing restarting over and over again bug). I was trying to figure out what the heck is going on so I checked few vids but it was hard to pin-point the problem until I watched full video file. Then I discovered that video had small jumping occuring every some time that was caused by server-end keeping real timestamps while it couldn't process video on-time making the stream drop and causing still frames or time jumping ahead in video player (when You watch already downloaded stream, if You watch it in real-time from the internet it usually hangs for a while or looks like broken I-frame).

So I started analysing videos by fast-forwarding with MPC and using ffmpeg to split and merge all chunks together. It worked pretty well at the beginning until I got stuck with it at some video and was really not up to the task of watching ~8 hours material with full attention.

After being disappointed with one of my vids that couldn't be processed by YT (and I really didn't want to process it myself just to deleted the file afterwards, worsen the quality even more or waste my time by it not being helpful at all), I started searching tha intarwebz and downloaded few programs that turned out to be shit (at least in my case, they may help for some, but they were all a waste of time for me). So I started digging into it by myself...

FFPROBE, some of You may heard about it, cool tool to scan video file and tell You stuff... well then, let's try reading frames... OH shit, 300MB of plain text file... That will take AGES to analyse and my eyes will most likely start bleeding after seeing it in notebad past first 10MB... but hey, there is a way around it... as I like automating stuff, let's try doing my best in this area.

So I opened notepad and started PHPing, vua la, a first automated script to get rid of bad video parts and YT accepted my video! A happy day!

But as I stared with C on my zopfli fork project and was quite successful with it to extend nice Deflate program functionality to even support ZIP files, I thought that I can try my best here too. Opened notepad and started rewritting PHP into C and then successfully GCCed it. Works perfectly! (at least in my sandbox!)

So I wanted to share this solution with You. It's available at http://virtual.4my.eu/KrzYVideoFixer/.

You can find there windows and linux (both x86 and armv7-odroid) builds.

Make sure to read readme.txt file and put ffmpeg and ffprobe in the same directory and run in from that directory (or to be accessed globally, like PATH variable on windows). I was thinking about adding switch to point ffmpeg/ffprobe dir manually but..... is that really needed??

The program is quite simple, takes ~500KB of RAM by itself, uses ffprobe to analyse stuff by itself and split damaged file to chunks then merge them together using ffmpeg. It DOES NOT convert anything. Stream copy is used all the way there, so it's pretty much the original stuff with few damaged parts being cut out of file. However, the output is always saved in MP4 format.

AVCONV WILL NOT BE SUPPORTED. Why? Because it doesn't want to work the way it needs to (avprobe is shit,,,,,, whoooops!!11!), just compile ffmpeg if You are using Ubuntu.

On the other note. I managed to successfully compile mf2t/t2mf (midi file to text / text to midi file) using modern gcc on windows and linux. In case You were searching for one, there is x86, x64 (both windows and linux) and armv7 (linux) versions there! http://virtual.4my.eu/mf2t. As far as I tested, after some sourcecode fixes to properly compile with GCC and actually run as it should, it worked pretty well even with SYSEX data. However, it may have bugs because I'm not that sure if I managed to fix all the code that required fixes. I will maybe add some functionality to it at some point when I have time... I'm personally interested in a switch to always start from tick 0 by shifting all ticks so the resulting MID file starts RIGHT AWAY; a away to make file that is ending too fast to have bit more seconds at the end with a switch for it; aaand somehow to detect/apply proper start/end timings for midi to loop nicely (if it's enabled in, let's say, in_midi... caught... winamp) with a switch.

If You would want to reward me for my work, just process some file with kvf and read summary message or join www.gptchat.com and be interested in some deals there. :)

So suming it all... buuuuuuilds...

KrzYVideoFixer: http://virtual.4my.eu/KrzYVideoFixer (linux x86, armv7; windows x86)

MF2T/T2MF: http://virtual.4my.eu/mf2t (linux x64, x86, armv7; windows x64, x86)

check readme first!

Thanks!

Zyxel NSA-210 successfully running 4TB Seagate 4000DM000

2014-04-26T01:29:00.000+02:00

This walkthrough will tell You how to make Your >2TB work in Zyxel NSA-210 with newest firmware when disk initialization fails. It does not contain images currently, maybe later. Maybe hard to understand and not enough detailed. I wrote it after I succedded and it was hard to remember all correct steps after 2 days of continous work on this issue. Also I consider that You have a bit of Linux knowledge that's why I skipped details here and there. Feel free to ask questions in comments.

Make Zyxel NSA-210 support Your 4TB HDD.

What will You need:

- Linux x64 (required, I used CentOS, can be in VirtualBox),

- USB->SATA bridge or desktop computer (required)

- Additional tools listed in this walkthrough if You don't have full installation of Linux (required)

- Patience (optional :)

- Clonezilla or other able to read EXT3 and XFS (optional)

- >2TB HD (optional, that is if You have one)

- <2TB HD (required, can be old HD that You kept in NAS)

First You may like to boot up Your NAS with old disk and check if everything works. If You didn't boot it up ever, then You need to do it with HD that is less than 2TB, even old 40GB SATA will do, we just need to initialize it with tools that were shipped with the NAS (or newer ones available on Zyxel site to download. WARNING! MUST BE FOR NSA-210, even if program is the same, newest versions for other models will just not work correctly, even if it appears so).

So init the old HD in NAS with newest firmware so it boots up, You can start setting it up but it's recommended not to make any shares and users yet, let it have admin only (root for shares). You should use as little space HD as possible because clonning it (if You decide to do so) will take a while. If You didn't put 4.41 version of firmware during disk inition (in disk init wizard) then download it now from Zyxel site and update the firmware with web interface. It's important to have newest one due to 4TB disk support, even though You may have trouble making it work at all.

Turn off NAS and take out old HD, use USB->SATA bridge or 2.5 inch USB case (in case You used this size to init Your Zyxel - yes it's possible to put in 2.5 inch HD, it just needs SATA sockets), or connect it inside Your desktop if You have one.

Now the things can get tricky. But You will need linux anyway, clonezilla itself will MOST LIKELY NOT help You, even though it may appear that Your NAS read it.

Easiest way to move firmware data is to just clone old HD onto new HD either using device->device setup with clonezilla or device->image then image->device setup. It's recommended to copy whole harddrive. WARNING! MAKE SURE TO UNCHECK PARTITION RESIZING!!! If You fail in any of steps after this You need to start over doing this.

Alternative way is to make Linux read Your partitions and copy the data on old HD to Your home directory. WARNING! BOTH PARTITIONS MUST BE COPPIED IN SEPARATE DIRECTORIES AND ALL HIDDEN FILES, AND FILE PERMISSIONS MUST BE PRESERVED (all, even when groups/users IDs don't exists on Your linux box/virtualbox).

Access Your old HD and new HD with Linux x64 box or virtual guest machine. In case of using USB->SATA adapters and VirtualBox make sure You don't use USB 3.0 connection, it will fail to read the drive even when 1.1 mode is set in VB - just use 2.0 cable or connect to USB 2.0/ USB 2.0 hub. You need to run terminal as root then type: fdisk -l and using Your sharp eyes and brain to distinguish between disk names like /dev/sda, /dev/sdb, /dev/sdc etc. Now if You already know which one is Your old HD then note down starting and ending point of first partition and starting point of 2nd partition. If You used clonezilla You most likely have same structure on Your new HD. You may already know what I want You to do but STOP, DON'T CREATE ANY PARTITIONS ON NEW HD (or alter them when cloned).

We now need gdisk or parted, because we need to make MBR into GPT partition table on new HD - Yes new firmware reads it all right. Both solutions are valid but I prefer parted myself more, even though it's most likely harder to use but at the same time I know what I'm doing because all things are done manually. Doesn't matter if You cloned Your HD or are starting on fresh HD, Your data will be intact if You DON'T do any mistake here because basically what we are editing is main partition table and XFS file system can be resized pretty nicely.

So now run parted on Your new HD. For example parted /dev/sdc (if sdc is Your new HD THAT IS!). Type in print and confirm that You have cloned partitions or empty table. Now type in mklable gpt. If You had cloned HDs it will give warning information, answer Yes. Now we need to switch to sectors as the measuring unit, type in unit s. Now check Your noted start and end sectors from original disk back when You used fdisk. For me they were: partition 1 - Start: 63, End: 1028159, partition 2 - Start: 1028160, End: not important. Now I would type mkpart primary ext3 63 1028159. Ignore the warning. Now type print and confirm that start and end sector matches Your previously noted range. Also take a look at reported number of sectors by the disk. This is kind of importand because You can't go over that range when creating second partition for data, in fact, You must stay away a bit from the end due to one sector being used for GPT partition table backup and it doesn't exactly means it's the last. Let's make 2nd partition, I typed mkpart primary xfs 1028160 7814033472, and again ignore the warning. I got away by like 2000 or 20000 sectors away, don't remember, but it's not that important if it's that away from the end of HD. You can also name both partiton, first should be named firmware, second is second. Not sure if it's necessary, but I did that just incase.

quit parted.

If You cloned image then You can skip this place. If You didn't then You now need to make ext3 file system on first partition and xfs on 2nd partition. You can search the internet on how to do that. No special settings need to be used here, just make sure they are the same size as place alligned for partition. After making file systems You need to copy all previously backed up data from old HD to new HD or directly copy if You have both HDs connected. WARNING! ALL HIDDEN FILES, AND FILE PERMISSIONS MUST BE PRESERVED. WARNING2! Be sure that Your XFS partition has disabled lazy-count, mount the XFS partition and run xfs_info on the mountpoint. If it has enabled lazy-count then You must umount it and run command that looked like this in my example: xfs_admin -c 0 /dev/md127. I have suspision that if You don't disable it You may run into the same problem as me, disk full messages after writing ~100GB to the HD.

This step is for people that used clonezilla previously. Now You need to mount XFS file system. You need to add -t xfs in the line since system may not detect it automatically. Now let's resize it by typing: xfs_growfs mount_point. It may take a while in case of big HD so go make Yourself a cup of tea or coffee. :)

After this confirm that all files are still visible on both partitions and umount them.

Now You basically should be ready to put Your new HD into Zyxel? Go on and try it... You will be susrpised that all this work is NOT YET ENOUGH! Why? Well You may have noticed something when running fdisk -l on old HD. There is some additional information under old disk. Yes, that's right. A RAID-Linear volume. Why the hell would the use RAID in one HD Zyxel? No idea, but I think it's to avoid errors when booting with USB drive connected, or to show more volumes if there are any (most likely it's possible), because Zyxel doesn't look for sdXY when booting, it searches for MD0 only for data storage.

How the heck then we make RAID stuff. Especially that we just made partitions and checked that data is intact. Well, it won't hurt Your already existing partitions if You do everything RAID, I mean right. :)

First we would like to read all RAID stuff on old HD, by typing mdadm --detail /dev/mdX where X is the thing that fdisk -l reported for old HD. For example my output looked like this:

[root@localhost media]# mdadm --detail /dev/md127

/dev/md127:

Version : 0.90

Creation Time : Tue Apr 22 18:36:51 2014

Raid Level : linear

Array Size : 155774208 (148.56 GiB 159.51 GB)

Raid Devices : 1

Total Devices : 1

Preferred Minor : 127

Persistence : Superblock is persistent

Update Time : Thu Apr 24 11:50:17 2014

State : clean

Active Devices : 1

Working Devices : 1

Failed Devices : 0

Spare Devices : 0

Rounding : 64K

UUID : 8296b7ff:c19f6ddc:7c88728c:d0f54ce9

Events : 0.29

Number Major Minor RaidDevice State

0 8 18 0 active sync /dev/sdb2

So we now need to create similar thing for new HD, having in mind that most of the info must match and we need to use MDx that is not yet used. For this I ran this command:

mdadm --create --verbose /dev/md0 --level=linear --raid-devices=1 --rounding=64K /dev/sdc2 --metadata=0.9 --force

It may be important to use 0.9 version metadata otherwise Zyxel may fail to see the volume. After running this command You are basically ready to put Your new HD into Zyxel. What is left is to loosly compare permissions being the same as the old HD ones by looking at few files/directories on both drives. ls -la will as well tell You if new HD consist of hidden directories/files and symlinks.

After this You may try to put Your HD into Zyxel NSA-210 and You should see volume of 3.64TB or similar size in case of 4TB drive. That is Your new data storage. If for some reason Zyxel appears unstable, like adding shares doesn't work or it can't shutdown, then reconnect it to Your linux box and run xfs_check or xfs_repair. Also I noticed that I needed to reboot NAS after adding a share, even though it already existed when I rechecked with my Linux box. So bigger HD may still have some problems with this old device, but it's yet up to test... Finally my 4TB Seagate 4000DM000 has nesting place to rest in. :)

So basically what we done was:

1. Created GPT on new HD (instead of MBR table as on old one),

2. Restored parition structure on new HD,

3. Resized data partition to cover whole free disk area,

4. Coppied all data from old HD to new HD including hidden files and permissions (or cloned it with clonezilla),

5. Recreated valid RAID-linear array for second partition,

6. Confirmed everything is ready to boot and run on Zyxel NSA-210.

I hope I helped. :)

~~~~

Mr_KrzYch00 ( http://www.4my.eu )