AMD's gaming super-resolution patent answers a lot of questions we've had about the Red Team's upcoming FidelityFX super-resolution (FSR) whenever AMD is ready to talk about potential answers to DLSS.

We looked over the patent application published today (via Videocardz), which was actually filed in November 2019. As far as we can discern from the patent terms and the often confusing back-and-forth between Jacob and I, this implementation of AMD's super-resolution feature should be platform-independent and could work with any game, not necessarily on any particular GPU.

If this is indeed the case with features such as AMD's DLSS-a, it would cover a much wider range than Nvidia's more restrictive technology. However, by being trained on a game-by-game basis rather than in a "fully learned environment," DLSS may yield more accurate results in specially coded games. However, the patent suggests that AMD's version of that approach may be rather impressive in its own right.

The more one hears about AMD's FSR, the more it sounds like the supersampling equivalent of the FreeSync vs. G-Sync battle; AMD's technology is more open and works quite well; Nvidia's is a bit more open and works better, but it's not as good. Nvidia's technology, on the other hand, is designed to be closed, but potentially a bit more effective.

Ultimately, if AMD FSR is good enough, it will become a much more useful and more widely used technology. That would be very exciting.

Anyway, back to the details of this patent application: what makes FSR seemingly different from DLSS is that it uses multiple upscaling networks (both linear and nonlinear) to try to retain important visual data from the original image. It also aims to "efficiently super-resolve images in a convolutional and generalizable manner." That would seem to mean that the neural network does not need to be trained on a specific game before being used.

The patent describes the basic flow of this gaming super-resolution as taking a low-resolution image (which can be quickly rendered by the hardware in question) and using two deep learning networks, linear and nonlinear, to downsample it into several different images The details are described in detail. These different downsampled images allow FSR to extract a number of different features from a low-resolution starting point, such as colors, objects, curves, etc., and finally use them to create a more detailed and accurate high-resolution image.

"The combination of linear and nonlinear upscaling is a combination of linear upscaling's preservation of color and larger scale features (large objects and shapes that are more easily recognized by the human eye) in an image and nonlinear upscaling's preservation of more It facilitates both preservation of detailed features (e.g., curvilinear features and features that are difficult to recognize at lower resolutions) through linear upscaling," the patent states.

"Linear operations use only input data, while nonlinear operations use both input data and other data that augment the input data (i.e., non-input data). Nonlinear functions facilitate the accurate determination of complex features (e.g., curves) of an image more efficiently than nonlinear functions (e.g., convolution operations)."

Multiple downsampled images are further pulled together to create pixels of a high-resolution image using a "pixel shuffle" stage. To do this, multiple inverted images are used to create a 9 x 9 pixel block for each low-resolution pixel, using eight shifted images and one unshifted image of the downsampled image.

After this pixel shuffling process, several more clean-up operations are performed before the final high-resolution image is created.

It may seem like a lot to perform at each frame, but when you look at what you expect your graphics card to do when throwing a game, it always seems like it.

While this is only a simplified process, the patent details that the super-resolution feature can use any or all of these elements in any combination. This may suggest that it can adapt to the hardware provided when used on different types of devices. It is claimed that it can be used in "computers, gaming devices, handheld devices, set-top boxes, televisions, cell phones, tablet computers.

It also states that the parallelism required to process the linear and nonlinear portions simultaneously can be achieved, but does not necessarily have to be performed on a GPU." If the hardware does not support parallel processing, "the linear and nonlinear upscaling processes will not run in parallel.

It even states that super-resolution can be performed in software, but inevitably it will be much, much slower, possibly negating the benefit.

Still, it sounds pretty exciting, especially if it can really run anything. If it could be made to run effectively on older graphics cards, or possibly even Nvidia GPUs, it could be an incredibly powerful tool. But that depends on how FSR is enabled. [but it could be integrated into games and game engines, like some of AMD's current FidelityFX features.

Perhaps both. It is possible that a game-independent version may exist in the AMD graphics driver, or a game-specific version may be enabled for each game.

As mentioned at the beginning of this patent screed, the publication leaves many more questions to be answered. At the beginning of next month there is Computex, where Dr. Su will give his keynote address. If rumors of a FidelityFX super-resolution release in June come to fruition, these two would be the most likely candidates for unveiling.